WBD375 Audio Transcription
Lightning Series: Privacy and Security with Christian Decker & Carla Kirk-Cohen
Interview date: Wednesday 21st July
Note: the following is a transcription of my interview with Christian Decker & Carla Kirk-Cohen. I have reviewed the transcription but if you find any mistakes, please feel free to email me. You can listen to the original recording here.
In this interview, I talk to researcher Christian Decker and infrastructure engineer Carla Kirk-Cohen. We discuss the current state of privacy on Lightning, potential vulnerabilities and most likely solutions.
“Lightning is built on top of the main chain, so you are bringing your on-chain privacy with you, although there’s a transactional increase in privacy...if you send a UTXO from Coinbase to fund your channel, Coinbase knows you sent that to fund your channel.”
— Carla Kirk-Cohen
Interview Transcription
Peter McCormack: Christian, hello, mate. I haven't talked to you for, I think it feels like about two years.
Christian Decker: Yeah, it's been about two years now, I think; yeah, two years.
Peter McCormack: How are you keeping, man; are you well?
Christian Decker: Yeah, quite good, quite good. It's been busy, but interesting, so can't complain.
Peter McCormack: Great. And, Carla, let's be honest, we've met before but I completely forgot about it; I'm so sorry. Remind everyone where we met before, because you've been on the podcast before.
Carla Kirk-Cohen: Yes, very, very briefly, although I was smiling so much at the time, I don't think I could almost get a word in! I did a little guest appearance at the end of John Newbery's episode with my fellow Chaincode residents, Amiti Uttarwar and Fabian Jahr, just to chat a bit about the experience of doing the long-form Chaincode residency, which is also where I met Christian; so, that's a nice little close circle, yeah. And, yeah, it's also been about two years since we spoke briefly.
Peter McCormack: Yeah. I remember that 15 minutes where the three of you just sat there with these massive grins, like the happiest people ever, in New York, getting to hang out in New York --
Carla Kirk-Cohen: Yeah, those were some happy days.
Peter McCormack: -- working on Bitcoin. But now, you're an old hat; you're a proper Bitcoin developer now!
Carla Kirk-Cohen: No, I wouldn't call myself proper, but yeah, I'm working at Lightning Labs. After my time at Chaincode, the excellent tuition I got there from Christian and others, I took a role with Lightning Labs, where I work on their Lightning Loop project mostly, and then also do a bit of work in the LND implementation.
Peter McCormack: All right, well before we kick off with what we're going to cover today, because we're going to be talking about Lightning and privacy, I think you should tell people a little bit about the Chaincode residency you did. Just explain what it is, because some people wouldn't have heard that show, they won't know who Chaincode are, what the residency is about; but I think it's a really interesting thing just to let people know.
Carla Kirk-Cohen: Sure. Chaincode is a magical place. It's an office in New York where Bitcoin developers get to go and work on free and open-source Lightning and Bitcoin software. I got to go there in 2019 with a group of about ten other developers, where they were running what they called, The Chaincode Residency. So, we were really fortunate. It was about two weeks of seminars and people like Christian, Fabrice, René Pickhardt; a bunch of Bitcoin and Lightning developers came in and gave us a great overview of the network and the system and the protocol and how everything works.
Then, we were the first group that got to do this and unfortunately, because of the pandemic, the last. We also had a glorious two-month period where we just got to stay in New York, work on projects that we were interested in and really dig into Lighting and Bitcoin deeply, which is where I first started really meaningfully contributing to LND. Now that the pandemic's hit, Chaincode has been running their seminars online, which I think is really great.
A colleague of mine who's just started, Elle Mouton, went to the Lightning seminars and now she's working at Lightning Labs, so I can advertise the remote version very highly as well. They're open on GitHub, they do run them every few months. You get into small groups, you do calls, you discuss all the content from our original seminars, because everything is on YouTube, everything is open source; and then, you discuss with a group of your peers. So, I think it's a really, really great programme. Adam Jonas, over at Chaincode, has just moved mountains to get people into Bitcoin dev.
The other thing I'll mention, just because it's also something I'm involved with, when I was at that residence, I got to meet John Newbery, who's now started an organisation called Brink, and they're also looking at improving the Bitcoin developer funnel, specifically for grants to developers. A lot of devs have funding from Brink to open projects, and with what I think is a wonderful programme, called the Fellowship Programme, which Gloria Zhao is currently in, and that's really just an extension of this Chaincode residency. It's a year; you go join them in their office in London, and you onboard onto Bitcoin Core. And, I am one of the members of the board of Brink, where we take applications for grants of fellowships; and I also think that's a great extension to the Bitcoin developer funnel, which is very much inspired by, and of the same quality, as the Chaincode residency.
Peter McCormack: Amazing. Yeah, I love the work John Newbery's doing there. We made a small donation. Hopefully next year, we can make a bigger one, but I love what he's doing there. So, you're at Lightning Labs, which is amazing. Is any of your code now out in the wild; are people using Lightning Network and your work that you've done, deployed code that's being used?
Carla Kirk-Cohen: Yes. My code, in my opinion, made its way out alarmingly fast. I felt very new and when I arrived, I was immediately digging into adding. So, my first big change, I guess, was adding a feature called Upfront Shutdown, which helps you close out, in a happy case, helps you close out your channels a big more safely in Lightning. And, it was deployed after a few months of review, because we don't add new contributions lightly, but it has been out for a while, since about LND 0.9.
Peter McCormack: Amazing. Well, it's great to catch up with you. I've interviewed Amiti as well, which is awesome, so it was great to catch up with her. Christian, it must be great for you as well though, seeing these people come through the programme and onboarding and just out there, contributing to the network?
Christian Decker: Oh, absolutely. It's always a joy to be able to speak to people about the more obscure details of the protocol and then seeing them join the community and help out, because that's ultimately what we need in the Bitcoin community, more people that are knowledgeable, that are able to contribute to this project, and to basically then surpass us eventually, to come up with new and interesting ways of doing things. So, we're definitely training people to replace us.
But the whole project also has a nice side effect of us basically being able to meet in real life and talk people through things, both as instructors, but also between specification people as well. So, we took the opportunity back in 2019, in Zurich also, to work on the specification itself. So, Fabrice was there, I was there, René was there, so we had a lot of very interesting discussions that proved to be very fruitful for the specification development itself. And of course, they had very simple implementations as well.
So, it's an awesome project and I would love to resume the in-person meetings again. There's always interesting stuff that comes out of these meetings, and they're just huge fun. So, if anybody wants to be an instructor next time, I can only highly recommend it.
Peter McCormack: Well, I want to get back to the in-person interviews. I've done a few. I did a few over the last couple of months when I was travelling, but I want to get back to in-person, because it is a lot better. We did two shows, I believe, when I did my Lightning series, which was a couple of years ago. And at the time, Christian, I was still a little bit like, "I've got no use for Lightning. I mean, I understand why it's important and it's cool that it works, but I just don't need it".
But my trips back and forth to El Salvador completely changed that, because pretty much the majority of people that I meet there, they are using the Lightning Network. There are very few actually using the base chain. All the vendors who are accepting Bitcoin in El Zonte, whether it's the pupuseria, the coffee shops, they all just want Lighting payments, and it was a real game changer for me. I realised there are places where Bitcoin is being considered as a medium of exchange.
So, I was talking about this and René got in touch and said, "Pete, you need to touch on this subject again. So, I've already recorded three shows; I've had Jack Mallers come on and shout very loudly about Bitcoin, which is always a pleasure; I did one with Andreas and René. I'm going to do another one with René talking about his Pickhardt Payments; and I did another one with the RaspiBlitz guys, talking about nodes. But today, we're going to get into privacy things. As per Christian's suggestion, Carla's going to lead the way on this and Christian's going to dip in occasionally. And I know, Carla, you're very excited about doing this.
So really, I think a good starting point is for you to start explaining the state of privacy on the Lightning Network, and why the Lightning Network is more private; because, someone like myself, I've always struggled with trying to achieve privacy on the basechain, there are so many things I have to think about. I have to think about my browser, I have to think about my wallet addresses, I have to think about CoinJoins; there are just so many things to think about. I know someone of my level would just screw it up, so I don't really operate with particularly high levels of privacy using the basechain.
But when I talk to the Bitcoin experts like yourselves, I get this kind of explanation that when you're using the Lightning Network, it is inherently more private. So, probably a good starting point is to explain why it's more private and the state of privacy on the network.
Carla Kirk-Cohen: Sure. I think I'll start at a really high level, so the Lightning 101, which helps us out a bit with privacy. So, if we think about the way that we transact on the basechain, if you want to make a payment from your node, you sign the payment, the transaction, and you broadcast it. And, when you broadcast it, actually what you're doing is getting up on a soapbox and saying, "Hey, everyone in the entire world, I would like to make a Bitcoin transaction. Please will you tell everyone that you know that I would like to make this Bitcoin transaction so that the miners will find out about it and eventually, it will make it into a block?"
So, that's flooding the network with transactions, which you want to get into a block. And of course, once they get into a block, then they're on the public ledger, which everyone has access to. And then of course, we know that companies like Chainalysis, that will take that public ledger and examine it and correlate the UTXOs.
Peter McCormack: Aargh!
Carla Kirk-Cohen: Yeah!
Peter McCormack: Fuck them guys! Sorry!
Carla Kirk-Cohen: No, no; reaction justified. But they will try and take this public ledger and they will say, "Okay, how can we identify individuals from this information we've got?" And then, the other thing they'll do, which I don't know for certain they're doing it in practice, but because of the nature in which we spread these transactions through the network, they can in theory spin up a bunch of Bitcoin nodes and connect to as many Bitcoin nodes as they can; and they just need a few servers to do this. They're such a big company, they have all these ridiculous government contracts, I'm sure they have money for it, to connect to all of these nodes.
So, when I get up on my soapbox and say, "Hey, everyone, I want to broadcast this transaction", I'm the first person that sent this from my node they're connected to. So, they can try and trace you within the network and within the way you actually broadcast these transactions. And that's actually a lot of what you chatted to Amiti about on a podcast, and Amiti's really passionate about addressing, the way in which we broadcast on the basechain, to make that a bit more private.
Then, when we take it into Lightning, we have a very different way of transacting. We don't do this big flood of transactions through the network; we do it in a much more private way. So, the first thing we do is we open up a payment channel. I know you discussed this with René and Andreas, but there's one transaction on chain which opens this payment channel. Once we have this payment channel open, we can use that to send and receive on the Lightning Network.
But when we do send and receive using that single on-chain output, we don't tell everyone in the Lightning Network that we're going to send this transaction, because then we wouldn't really be scaling; and part of the point of the Lightning Network is to scale. So instead, we wrap everything up in a very nice onion and send it through one route in the network, so only the people who are on path in that payment, so the people between me and the people I'm paying, can actually see that this payment has occurred. And, even though they can, it's onion-wrapped, so they can only see who it's coming from and who it's going to.
We can dig into that a bit more later in the podcast, but essentially what this means is that senders get really good privacy, because no one can see where it's going; and, receivers get some improvement in privacy, although we still have some quirks to work out there. So, once you've opened this channel on chain, you send and receive and you can do all of these transactions, and you can route and all of these things; and when it becomes time to return to the chain, the only thing that closes out on chain is the aggregation of all of those transactions, so it shows a change in balance in that channel.
So, say I started with zero and you had a Bitcoin; maybe when we close out, we each have 0.5 Bitcoin. So, there is knowledge that a change in balance has occurred; but we could have sent 0.5 Bitcoin between ourselves, we could have sent 1,000 transactions back and forth, we could have just had a payment routed through our channel and actually not interacted with each other at all. So, that level of aggregation that we achieve is really nice in that there's not so much information on the public record. And on the network itself, everything is onion-wrapped and so it's much more private.
One thing, I think, when you mention it's a bit difficult for you to manage your UTXOs, one thing I think that we do leave behind a bit when we talk about Lightning privacy, and Anthony Ronning's got a great article that he put out a few months ago on this, is that Lightning is built on top of the mainchain. So, you are bringing your on-chain privacy with you. Although there's a transactional increase in privacy, so when you send and receive you have better privacy, if you send a UTXO from Coinbase to fund your channel, Coinbase knows that you sent that to fund your channel.
Those channels on the Lightning Network are announced to everyone in the network, so that we can route, much like we announced those transactions, which has a bit of a privacy issue, because you announce your node's public key and your node's IP address in most cases, when people aren't using Tor. And there, it gets a bit tricky, because suddenly we have UTXOs, we have IP addresses and we have publicly broadcast information, just like the transactions on the basechain, and companies could maybe come and try and learn this information and use it to cluster and de-anonymise people, like they do on the basechain.
Christian Decker: Yeah, I think that's the perfect difference here. The difference between the groups that you're actually telling about a certain transaction happening is quite limited when you talk about Lightning; whereas in Bitcoin, well it's a broadcast, you're telling everybody. But there's another part to this, which is the difference between the information being persistent in Bitcoin; because chain analysis companies can basically come in much, much later and analyse the blockchain, and because the transaction you've published to the blockchain has persisted, they can basically work backwards from the current state and uncover your profile, even afterwards.
So, if you thought you had perfect privacy right now and sometime later there is a discovery that makes it possible to create a profile on top of the blockchain, you might actually uncover the past profiles of users as well, not just the ones that are currently in the network. This is in stark contrast with the ephemeral nature of Lightning payments, because the payments that are performed on a Lightning basis are only shared with the involved parties. But this information is also forgotten as soon as the parties go away, so there is no way for a chain analysis company to basically go back in time and look at a persistent trace and then build profiles on top of that. So, there's this persistent versus ephemeral nature of information.
When it comes to actually looking at the transaction propagation that Carla mentioned before, where a chain analysis company can basically spawn a number of nodes and look how the information is propagated in the network to identify roughly from which geographic location a transaction was started, that is not hard at all. I did that during my PhD during the first year. You open a couple of hundred connections and you see it can then trace where a transaction basically is transferred on the network itself. So, I wouldn't put it past them to actually be doing this, unless they're lazy!
Carla Kirk-Cohen: Arguable!
Peter McCormack: Well, my expectation is Chainalysis, Jonathan; is his name Jonathan Levin? I'm trying to remember that guy's name, the guy who likes working for governments; my expectation is they're doing something. They're trying to see how they can get data out of the Lightning Network. I'd be very surprised if they weren't.
So, I do have a few questions, Carla, I wrote a few things down here. Firstly, look, I know what onion routing is, but there's going to be people listening who won't have any idea what you're talking about. Do you want to explain what onion routing is?
Carla Kirk-Cohen: Yes. So, on a very high level, and thank you to Christian for drawing many pictures on the board at Chaincode to really get me through this one; but in short, if I want to send a payment to Christian and it's going to go from me to a couple of different parties in between, instead of doing what the internet does and saying, "Hey, guys, I want to send a payment to Christian. Here's Christian's address; please, just will you do your best to send it", and everyone on the internet does what IP forwarding does, it does look up his address and try and send it in the general direction of where he is. And, that's not very good for privacy, because then every single person who sent this packet along the route knows that I was sending it to Christian and they can quite easily look back and see, "Okay, this was propagated through this route".
Now, the way we do it on the Lightning Network is different, because instead of saying, "Okay, here's my end destination", I, as the sender, look at the Lightning Network and I try and figure out the route I'm trying to take to Christian. I know you've discussed this in a previous pod, so I won't go into it too much; but once I've created that routing path, I actually take each hop and encrypt it with a key that belongs to the hop on that route. Then, I take them and I put all of them inside each other and create what's called a very big onion. Then, I send this to the first person along the route.
Then, you only have the first key, so they use that to decrypt the first layer of the onion, and they can see that it was sent from me, and they can see it needs to go to the next person. But they don't know whether there were any hops before me, and they don't know whether the next hop is the last hop, or are there many more hops after. We also do a really nice check in the Lightning Network where we actually re-wrap up this onion to keep it the same size, so you can't get a really small onion from me and say, "Oh, look, this clearly is going to terminate at the next hop". We keep it at a constant size as we forward it along.
So, it takes every little hop and it's unwrapped; you know where it came from; you know where it's going; but you don't know anything in between. It's the same thing that Tor uses.
Peter McCormack: Okay. Probably a question for you, Christian, in that we know some services block CoinJoin UTXOs. Do we ever worry that there'll be a risk that UTXOs that have been associated with creating Lightning transactions may eventually be blocked by certain services, because it's seen as a way to gain privacy? It seems like any service in Bitcoin that is used to gain some form of privacy becomes something that regulators are scared about.
Can you identify UTXOs that have been used for people transacting on the Lightning Network; or, is it just something that's created on the basechain and nobody knows?
Christian Decker: That currently mostly depends on your own usage of the Lightning Network. So, looking at the footprint that a Lightning channel leaves on the blockchain, it basically is just a multisig. There are a number of good reasons why you would want to have a multisig, for example to secure your own funds, to basically have shared custody over some funds, and Lightning is but one use case for that.
Now, when you close a channel, then we have a second footprint on the Bitcoin blockchain, and that might give away some information about whether that was a channel or not. In fact, I myself am collecting this information to basically see how the network grows and how the performance of the network is, to better inform our specification approaches on the success probabilities of channels being closed correctly or incorrectly, or things like that.
So, if we mutually agree on closing a channel, then there is very little telling an observer that this was actually a Lightning channel. This is simply because we, when we set up the channel, we created a 2-of-2 multisig output and when we close the channel, we just spend that 2-of-2 multisig. There is nothing special about this closing transaction; it just looks like any other transaction. So that's, let's say, the happy path.
Now, if we start doing things like unilateral closes, because our counterparty went away and I need to get my money out of this channel, then there already are some indications that this was indeed a channel and specifically, there is a timeout involved and a script structure that will tell an observer, "Yes, this is very likely to be a Lightning channel". Even worse, if we have a cheating party, then it becomes very clear that this was a Lightning channel, because the counterparty will retaliate against this cheating attempt and will claim all of the funds in that channel. So, the more collaborative we are, the fewer traces we leave on chain; and the less cooperative we are, the more we're giving away our privacy.
This could be used in future to actually say these funds are tainted or not, but it is basically a race between us and whoever is trying to enforce these things. If we manage to taint all of the coins by being, at some point, involved in a Lightning channel, then they'd be out of business if they were to do something like this. The more we normalise the use of Lightning, the less these bad actors can actually use that information to say, "Oh, no, this is bad money that should not be accepted".
Furthermore, in future, we can actually improve that using Taproot and Schnorr signatures. So, at that point, we won't even see that these are 2-of-2 multisigs; it will just be a normal transaction that looks like any other transaction. And so, with these incremental improvements, we can also improve the privacy of the channels themselves.
Peter McCormack: Okay. We're going to get into some quite technical details and that's where I have to be honest that Ben, my producer, helped me prepare some of this, because a lot of it goes over my head; I'll do my best to understand it! But, Carla, my basic understanding is okay for the next question, in that what I understand about Bitcoin is that there can be multiple implementations, but most people are using Core.
What is the state of Lightning? I know again there can be multiple implementations, but are people mostly using the Lightning Labs implementation, or is there a bit more of an even spread?
Carla Kirk-Cohen: I don't have very up-to-date numbers with that, but I think the usage paper came out that about 65% to 70% of the network is currently using LND, because in terms of privacy, when we use these different implementations, there are slightly different signatures that all of us leave, be it a different default value; or recently, I saw something that c-lightning sets some funny alias name and you can figure out that a node is a c-lightning node if it uses one of the words on the alias generator list. So, there are these really small things that having different implementations will affect privacy.
If people start to set their own defaults and really manage nodes quite differently, I think that will change. And, in terms of the comparison with Core, I think that the reason most people use Bitcoin Core first of all is because the code sort of is the protocol, whereas in Lighting, we have the BOLTS and we all do our best to abide by them, so we have a much more formal consensus system, even though Lightning doesn't need global consensus, which is why it's much easier for us to have different implementations versus Bitcoin Core, where you if you have a break from consensus, it's actually pretty catastrophic for everyone who has been broken off from the main network.
Peter McCormack: Christian, just on that, just to add to that, are there any particular risks; I mean, I honestly don't understand stuff like this, but are there any particular risks with multiple implementations whereby, I know these are interoperable; but when I get the explanations of how the Lightning Network works, there's a lot of complexity behind there, there's a lot of complexity that's been built into the system to make it work. Are there any particular risks in having multiple implementations that things aren't interoperable; there are mistakes that happen?
Christian Decker: Yeah. I think the key differentiation between the base layer and Lightning lies exactly in how big the group is that needs to reach consensus. When we're talking Bitcoin on chain, then we're talking about tens of thousands of users that all need to agree on a certain value, and where a slight misinterpretation of the rules might actually end up splitting the network in multiple parts that are unable to reconcile at any point in future, which is a catastrophe when it comes to a monetary system of course.
We've seen these kinds of things where the description didn't match what was actually happening. In 2011, I was actually trying to build my own implementation of Bitcoin, so we had to go and --
Peter McCormack: Me too!
Christian Decker: We actually had to go in and look exactly how Bitcoin did what it did; and we found a couple of discrepancies where the code definitely did not do basically what the label on the tin was saying, like the op check multisig operator which, for some reason, took one argument more. And, if you didn't implement that same bug in exactly the same way, then you would come to a different conclusion than Bitcoin Core would; and suddenly, you wouldn't be in agreement anymore with the rest of the network. So, you'd basically be inadvertently forking off the network.
This is then compounded by the fact that, even if we have a very detailed description of what the protocol does, there is no way for us to actually verify that multiple implementations do the same thing in all cases. So, that's a research topic that has been worked on in computer science for decades now. So, I don't think we're going to solve it quite easily right now.
Now, when we talk about the Lightning Network, this is completely different. The group that has to come to an agreement is the two of us. So, worst case scenario, if we have an incompatibility, then it's just the two of us that might end up in a stalemate; and we also have the backup mechanism of going back to the Bitcoin chain and letting the Bitcoin blockchain resolve this conflict, or difference in opinion. So, we actually have this way of dropping back on chain.
Funnily enough, that's what we do in most cases is, "Oh, we don't know the current status. Let's have the Bitcoin blockchain resolve that for us". And, the Bitcoin blockchain does not, of course, have this backup system, so it needs to be absolutely bulletproof. So, there is a difference both in the size of the consensus, but also in the backup systems that we have available. Therefore, for Lightning, we chose to go for a much more specification-driven effort, where we write down how things should be working and then based on that, we can build multiple implementations that adhere to this specification.
If two implementations disagree, well then we go back to the specification and can say, "Okay, this is how it should have been done. This implementation did it wrong", so we have a specific set of actions that we can go to and fix this. And, if the specification doesn't say anything about this, then we need to patch that hole and make it more precise so that any future implementations will not run into the same issue.
Peter McCormack: All right. Well, we're going to get into some of the more technical stuff now, and as we go through this, Carla, one of the things I want us to do is, I like to use Bitcoin without having to think about too much of the complexities. I like to use it like I use PayPal or Visa, or any of it, and just not even think about it. So, when I go to the shop and I buy something, I just use my ApplePay or scan my card and everything just works; I don't have to think about everything in the background.
I kind of want the same with Bitcoin sometimes, right? If I need to send Christian $1,000 of Bitcoin and he sends me a Bitcoin address, I just send it and don't think about everything in the background. When I'm in El Zonte and I'm buying a cup of coffee, again I don't really want to think about it; I just want to scan the QR code, or scan the invoice and pay it.
A lot of the time when things get explained to me about the Lightning Network and how different things work, I'm like, "Is this something I need to be aware of, or is this just how it works and you're explaining it to me, but I don't really need to care?" And, the reason I want to separate these is because we've got a range of people listening. There'll be someone like you, Carla, who understands all of this; so, when it's being explained, you'll get it all. There'll be other people listening who'll be like, "Holy shit, is this something I need to be aware of; do I need to understand how this works, the way these channels open and close and the rule sets?"
So, it's really good to give a guidance on whether this is something someone needs to be aware of, or if this is just how it works in the background; does that make sense?
Carla Kirk-Cohen: Yeah, of course. I think it's one of those things, like a 50/50 case of you always asking a bunch of nerds how these things work, so of course we want to talk about them; and some of the things people need to know about, some of the things people need to know about right now, but not always, so I'll try and make the distinction.
Peter McCormack: All right, well we're going to work through a list that Ben has provided me. Thank you, Ben Prentice; you're amazing. So, the first thing he's put on my list to talk about is jamming and HTLC spamming. So, I think it's good to explain what it is, how this works, and what you guys are doing to prevent this.
Carla Kirk-Cohen: Sure. So, when we talk about HTLC jamming or spamming, or whatever we call it, it is, on a very high level, if someone decided they wanted to grief the Lightning Network, they can send a bunch of payments with junk preimages; so, the secret that you use to claim the money on Lightning is called a preimage. But basically, they send a bunch of junk payments through the network, and they create all of these onions and they forward them through and maybe they can do this on a very high level and just send tons of payments to the whole network; maybe they can do this on a specific level and send all of the payments through your node.
The reason they can do this, it's a lot like the Bitcoin basechain back in the day before we had minimum relay fee and a bunch of these things, in that there's no cost when a payment fails on the Lightning Network. If you send that payment all the way through and it gets to the end and they decide to fail it back, there's no expense fee, there's no routing fee charged. And that, I believe, is quite intentional, because part of what's cool about Lightning is that you get to make micropayments, and it's also quite difficult to route.
So, if you had to suddenly start to pay every single time you failed to route, that's a terrible user experience. You don't want to go and pay for a coffee in El Zonte and have to pay a few cents because your payment failed; that's a very, very bad user experience. But because we don't have this cost associated with failure, if someone chose to, they could send a bunch of HTLCs through the network and lock up someone's liquidity. So, a channel only has a certain size; they could fill up that channel with HTLCs; or, they could just send a bunch of really small HTLCs, because the channel can only hold a set number of HTLCs, and then hold them for quite a long period of time, because in the bad case of a Lightning payment, we wait a few blocks to see whether it's going to resolve or time out. In the good case, it goes through near instantly.
But in that bad case, which they can exploit by sending a payment to themselves and just holding it for as long as possible, you can actually jam up some of the channels in the network. Whether anyone's actually done this yet, I don't think so; and, to the end user, if you're not running a big routing node, if you're not Fold, or Strike is using this on a big operational level, this is definitely on a class of things you don't need to care about. You would probably see, "I can't send a Lightning payment right now; that's quite annoying".
In terms of ways to address this attack, I think it's very similar to back in the day when people were having the big block argument and decided to spam the basechain with a bunch of really low-fee transactions. They decided to jam up everyone's mempool and for a while, the network got a bit stuck. But then, those transactions are cleared, the developers had a look at it and said, "Okay, we're going to introduce a minimum relay fee", so if someone chose to try and jam up the mempool, then they actually are going to end up paying this minimum fee. So far as I know, no one's ever done it again.
Ideally, we would find a solution to this before it happens, because it's never nice for things to break live and have to scramble to fix it; but at the moment, there are just a few different proposals as to how to fix it. We wouldn't want to add this cost to routing; it seems like it would really suck, it would be quite a bad user experience. We probably don't want to do something like rate limiting, because then an attacker can still degrade the network quite badly. So, that one's still a work in progress.
But in terms of the end user and needing to care about it, I was an econ student as well as a computer science student and I did a year of economics and I thought I was going to change the world with economics; so, thank goodness I found Bitcoin, because that was never going to happen. But I did this great course. It was The Economics of War, Terror and Crime and looking at people we label as criminals, terrorists, etc; and using these economic studies to prove that these people are rational actors. Although you think that someone that does these things might be crazy, they're actually doing something with a very specific cost benefit analysis in mind. They're making rational economic decisions doing what you might think is an irrational thing.
So, when I think about, maybe someone's going to jam up the Lightning Network with HTLCs, they are going to have to spend a bit of money to do it. Maybe they'll be able to tell their friends that Lightning sucks once they've done it, but there isn't really an economic incentive to do that right now; especially when we're in a world where you can just send someone a phishing email and they'll give you their seed. There's just very little incentive to make this -- if you go for the weakest target, you always do and it's a rational thing to do; and even if your objective is a grudge against the Lightning Network, because you're a Bcasher, it's much easier to make other attacks on people.
So, it's a theoretical problem that people are working on. Christian's probably got a better idea of some of the more up-to-date solutions for it, but I don't think it's an issue for the end user as it stands.
Christian Decker: Yeah, so one rationale for causing such an attack would, for example, be one routing provider trying to pin out the competition. So, there are some scenarios where this attack might actually be rationale; but in this case again, the end user would not see the impact, because basically it's these routing nodes fighting amongst themselves, but for their traffic.
So, they likely will even benefit from this kind of thing going on, because their routes might even become more efficient; there might be more redundancy; the network, as a whole, might become more resilient; and overall, the more we remove single points of failure that might be the target of such an attack, the more resilient the network as a whole becomes. So, if you are under attack, it might be bad for you; but for the network as a while, I think it might actually end up being a net positive.
As for countermeasures, there have been proposals basically to start unwinding the privacy a bit on a per-use case, so this is a proposal that Rusty had a couple of years ago where, if a payment is stuck for a long time and I basically look at my watch and say, "This is not good. I want to either make progress on this, or I want to remove it, because it's never going to happen", then I will ask the next person that I forwarded this to, "Hey, where is this stuck? Now, provide me with proof that you've done something to free these funds that are currently stuck, or I'm going to close the channel with you".
So, by doing this, you can basically follow the trail of the transaction itself, incrementally learning more and more about the payment, again just because it is stuck, basically; locate the place where it got stuck; and if it was a malicious attempt by somebody to hold these funds locked up, then we can actually punish them by basically causing them a loss, a monetary loss, in the form of a channel being closed, for example.
This is one proposal that has seen some work, but we've recently moved more towards a more proactive way of doing this, by basically imposing a cost, even to failed attempts, by having upfront payments. So, whenever I forward a payment to you, I will also have to transfer some money that is independent from the success of the payment to the next hop. By doing this, I'm assuming that the worst case is going to happen, and then there is a cost associated with a payment and basically, paying for that capacity that is being locked up, in the worst case.
Now, there are a lot of details in here. For example, if we get this wrong, you might always, as a routing node, be able to get the upfront payment and reject everything because, "Well, all I wanted was this small amount of money that I'm assured to get, and not forward any payment". And on the other hand, we need to make this payment proportional to the time that a payment might be stuck. So, me holding onto a payment for funds in these channels for a week should definitely cost more than me holding onto funds for two seconds that it actually takes to perform a real payment.
So, there are a lot of details that we have to discuss and have to take into consideration when designing these things, in order not to open new issues; still being fair and not sabotaging our actual use case, which is enabling fast payments using Bitcoin that are settled immediately; without weakening the privacy guarantees that we have; and without slowing down any of this and making it prohibitively expensive. So, yeah, we are working on these proposals in the background, but so far we've not found the golden solution that we'd like to move forward on, but I'm hopeful that we will eventually get there.
Peter McCormack: I'm pretty sure you will. Okay, so the next thing, Carla, that I find interesting that we're going to be talking about is probing and probing balances and senders, recipients, because that sounds like the kind of thing Chainalysis might try; they might be interested in looking at. Can you explain what's going on here?
Carla Kirk-Cohen: Sure. So, the concept of probing is pretty related to the idea of channel jamming, although you wouldn't do it at such an extreme level as trying to shut down someone's entire channel. But the high-level concept is that all Lightning channels that are public advertise their capacities. The sale of 1 Bitcoin in your channel to Christian; everyone on the network knows that you have 1 Bitcoin. But what they don't know is, is that 1 Bitcoin entirely on your side; has some of it been sent to Christian; what is the actual balance between the two of you in this channel? That's intentionally kept private.
The reason that's kept private is, in theory, if we were to advertise these balances all the time, you would be able to trace payments through the network; because, if you saw a 10-sats shift in my channel with you, then 10-sats shift in my channel with Christian, it's a fair assumption that I made a 10-sats payment to Christian, and then we both lose our sender and receiver privacy. So, that information is intentionally not revealed in Lightning, even though it would make routing very, very simple and probably reduce our failures quite a lot. We do it for privacy; there's always a trade-off somewhere with privacy if you look hard enough.
So, the concept of probing, if you were Chainalysis, and you say, "I would like to do this tracing of payments through the network, and the way that I will do this is by trying to find out the balance on a specific channel, and then apply the same attack to many channels on the network, trying to trace that 10-sats movement through the network". The way you can do this is, you send a few payments through a specific channel and you start with, say, the full capacity of the channel. If that can route, it means that full Bitcoin was in the channel. But if it couldn't route, then it means that somewhat less of that balance was available; so, you send a smaller amount. So, say, you send 0.5 Bitcoin. If 0.5 Bitcoin goes through, you know that between 0.5 and 1 Bitcoin was on your side of the channel.
You can use something called a "binary search" to send various amounts through to very quickly and very efficiently narrow down the amount that's in that channel, based on the failure you get, because we return a specific failure when we say, "I can't route this; I don't have balance". So, what Chainalysis is doing is, you're sending all of these payments through a bunch of channels and basically searching, trying to narrow down their balance. And, once you've done that to one channel, you need to also simultaneously be doing this to a bunch of channels, because finding out the balance in isolation isn't really an interesting piece of information. You need to see this balance change through the network.
That, in theory, is a way that you could try to de-anonymise senders and receivers. I'm not sure how much it works in practice, because when we change balances in the network, we do sometimes batch updates. So, if there was a change of 100 sats between me and Peter, it doesn't actually mean that there was a 100-sats payment; it could have been two 5s and a 90. So, even if Chainalysis were able to send all of these payments through binary searching and get the balance change to the network, I think it becomes a pretty complex task pretty quickly to try and divide this up into individual payments.
So, while you can certainly and incredibly trivially get the balance of one channel, we did that at Chaincode residency in under an hour, the ability to actually de-anonymise people and trace payments through the payment, I think will be a lot more challenging, because suddenly you need to match up all these different amounts, all these different payments, which is actually quite a computationally hard thing to do; unless there's only person using the Lightning Network, which seems pretty implausible now.
So, that balance attack does exist but again, the impact on the end user, I don't think they can say to the government with a very high degree of certainty that they've broken privacy on Lightning.
Christian Decker: Yeah, so I personally dislike the framing of probing as being an attack so much, because probing does have some very legitimate use cases as well. I mean, every time we perform a payment, for example, we send out attempts of payments and they either go through or they come back. And there is a certain point of view these are actually probes as well; we are probing whether we can use a channel for a certain payment or not.
This turns out to be a very important part of the payment process itself. So, we send out a partial payment to one side, and then we get it back, because one channel didn't have enough capacity; and so, us learning some information about the current channel capacity of certain channels that we might end up using is actually really helpful, because then we can split the payment in two and try two different routes. And, this is actually part of our payment process.
Now, I do agree that when you try to pinpoint the exact balances of a channel for the purposes of basically collating this information with what you know about other channels, and then tracing a payment as such is definitely nefarious, and we should do everything we can to eliminate this. And, in a certain sense, we are. For this attack to actually work, to identify payments, as Carla correctly said, we have update batching. So, if multiple payments go through the same channel, then they get batched into the same balance change and an attacker would then have to basically desegregate them again and sort of assign them, which is computationally a very hard problem.
So, the more traffic we have in the Lightning Network, the less possible it is for an attacker to actually identify this. As Carla said, when you only have one user in the network, it basically becomes trivial, because you can assign any change to that user. Now, if we have tens of millions of users, then the channels will be so busy that all of this information that you might be able to collate among channels is so noisy that you can't really identify a single source for this change.
In addition, we do have limits on the size of individual payments that we will forward, called the "HTLC's maximum size". So, if your capacity is above this size, then you will not be able to basically infer that, because the biggest payment you can ever make is basically 10, and if your balance is 15, we cannot go further than 10.
In addition, we do have some ideas of randomising this as well. If you collate information across multiple channels to basically rebuild what the actual payment is, by introducing noise in those measurements, you can throw off an attacker that is trying to make this system. And, even if they were able to sort of attribute every single channel change on the network to a payment, this noise would throw in enough uncertainty to make these determinations absolutely pointless.
A third defence that we have is basically, probing is not instantaneous. As explained, to probe a channel, you send one payment and you see it's too big, so you try a smaller payment. So, to probe a single channel, it will take a couple of seconds. Now, multiply that by the number of channels we have, and to get a complete picture to be able to infer the entirety of the network, that's a very, very compute and resource-intensive process, because you will have to have about the same number of Bitcoins just to probe the network as the rest of the network as a whole.
So, all you can ever dream to do is probe a part of the network, so anything that might start or end outside of that network, you will not be able to infer correctly. So, I think we do have a pretty good picture and the more users we get, the more unprecise these things get; so, I'm quite optimistic that this is going to work out.
Peter McCormack: It's starting to sound to me, Carla, like most of these are really theoretical attacks and not things to be hugely concerned about yet, but it's still great that people are working on it. I do want to ask about Flood & Loot and a fee blackmail attack, just because it sounds kind of cool!
Carla Kirk-Cohen: I think Flood & Loot does specifically fall into the theoretical attack, because it's a real apocalyptic version of something more specific that we're looking at in the spec, and that is the question of, can we get the transaction to chain within a certain number of blocks, because Lighting relies on this principle in many different ways? It relies on it for the penalty transaction if someone tries to cheat you; it relies on this to claim HTLCs, which are active in a channel, if you need to go to chain in the uncooperative case; and when that assumption that we can claim on chain in a certain number of blocks fails, there is the potential for loss of funds on Lightning.
So, what Flood & Loot looks at is a really extreme case, where someone decides that they want to attack the Lightning Network. The have to open channels and they have to send HTLCs, so they do have to commit funds to do this attack, although you can do it when fees are low on a Sunday night, so maybe not so many; but there needs to be capital present to route these HTLCs, because they need to initiate payments. They send a whole load of HTLCs through the network and then they just hold them, and they wait for a big time-out case to be hit. And, in the time-out case, this attacking node will not release the payments, so we will be forced to go on chain, because this payment can't resolve.
The theory here is that, if you have enough HTLCs that you can spread through the Lightning Network, you know when they're going to time out, so you can actually predict when you're going to congest the mempool, because it's often very difficult to know exactly when you're going to congest the mempool. But if you can spread enough of these HTLCs that will all go on chain at the same time, you'll have a bunch of Lightning nodes that now need to claim their HTLC on chain, but the mempool's full, because all these channels haven't closed. So now you, as the attacker, can come in and claim funds on one side and people don't get into the chain on time to make that base assumption.
I don't think the apocalyptic version of Flood & Loot is likely to happen. I think that we need to engineer solutions for a full mempool, because the Bitcoin mempool is going to be full; and, this is something that is really actively being worked on in the spec right now, in the individual case, because if we solve the individual case, assuming a full mempool, then we solve the Flood & Loot case. Because, it doesn't matter how the mempool was full; the mempool was full and we need to deal with it. That's why we have Lightning.
Peter McCormack: Anything to add to that, Christian, or did she just nail it?
Christian Decker: That's a perfect explanation. I mean, it is a very nice combination of using the Lightning Network to attack Bitcoin and then cashing in on the Lightning Network, but we've had to deal with these kinds of mempool congestion issues for a while. So, by addressing the underlying problem, we can make this much safer and basically, give the ultimate solution for all of this right away.
Peter McCormack: Well, the next question I wanted to ask you then really, Christian, is these potential attack vectors, are they all traditional computer science attack vectors that people were aware of, or distributed systems, or specific to Lightning, or just a combination of everything; because, if they're specific to Lightning, I'm assuming there are perhaps unknown attacks that people might be working on?
Christian Decker: I think they're probably a combination of both, right. We are often able to take what is seemingly a new attack in Lightning and basically, reconcile it with some existing theory in computer science; and then from there, use the proposed solutions that were proposed 10 or 15 years ago and make use of them to solve our actual problem in Lightning. But sometimes we do have really novel issues that we definitely might want to address.
One of these cases, for example, is the transaction pinning attack, which boils down to my counterparty being able to broadcast a transaction on my behalf; and might do so in a way that makes it really unlikely that this transaction is going to be confirmed. Similar to the Flood & Loot attack, that might then lead to me being unable to recover some funds and basically, the attacker being able to take the funds from me. So, this is a very novel attack that, as far as I know, we haven't seen in anything computer science-related so far.
So, this is also where it becomes very interesting for academics and researchers in general, because where other people get afraid, we get excited! We see something that we can dig into and try to come up with novel solutions and basically then, take something that is very scary and make it exciting and find a clean and nice solution. So, I find this very interesting. And, yeah, so far we have a couple of issues that are yet unaddressed, but they are mostly low priority, I would say. And, having the specification process where we have a lot of eyes on whatever happens in the specification and proposed changes, also helps us to identify these kinds of things very early on.
So on Tor we have, for example, found this transaction pinning attack and we have had some very constructive discussions internally, and that meant that a couple of proposals that would have fallen under this umbrella of this attack were discarded right away, because they would have been vulnerable. So, having many eyes on this and having many people working on this and many sorts of adversarial minds working on this is also a great benefit of the Lightning community.
Peter McCormack: Just bringing it back to how I use the Bitcoin Network, I operate an Umbrel node and I'm considering testing a RaspiBlitz and having a play with that. But I have a Lightning Node with my Umbrel setup. I'm not using it just yet; I still haven't fully played with it, just because I was away for a good eight weeks after I set it up. But am I at any risks myself with regards to exposing my IP address to my node; is there anything there, at a personal level, I should be doing?
Carla Kirk-Cohen: I think the risk with an IP address is that, if someone like Chainalysis were listening to the gossip that we send around the Lightning Network, there is a reasonable expectation that a government or surveillance company would be able to link your IP address to your real-world identity, because you pay a service provider; that service provider knows your name; they know your credit card. So, I think that is one of the big privacy considerations for Lightning, specifically when you have public channels.
When you have private channels, your IP address won't be exposed to the network, which is why I think Umbrel actually runs on Tor by default, as does RaspiBlitz, if I remember correctly. So, then you are covered in that your IP address is not being advertised and I think it's amazing that these node packaging solutions are actually going with Tor first, because it is a hard ask for a regular user to set up Tor and get going with that level of security and privacy.
So, I think you're okay. The only thing, if you're opening public channels, you still do need to be aware that the UTXOs you use could still be associated with that Tor address, because they are still clustered. But other than that, if you're using something which is using Tor, you don't need to worry as much.
Peter McCormack: That's typical. I had no idea that I had Tor and I was using it by default, but I'm okay with that level of my un-nerdy knowledge, because I kind of just want to be in that place where things just work for me, which I've always talked about. I'm not ashamed about it, but it doesn't mean I don't appreciate all the good work that people like yourselves do, or the Umbrel guys do, in getting this together.
I do have a closing-out question for you, Carla, but there's lots of things Ben sent me and there's lots of other things that I could have asked you and talked about; but is there anything specifically with regards to privacy on the Lightning Network I haven't asked you about that you wish I had?
Carla Kirk-Cohen: I think it is worth mentioning that the privacy risks for receivers on Lighting is slightly different to the model for senders. So, when you are sending, you get to create that onion yourself and you have very, very good privacy on Lightning. But when you are receiving, you do give someone an invoice which tells them how they can make this payment to you in the network. And, because they have to do this pathfinding and they potentially have to deliver payments over private channels, you do actually give them some information; so, specifically, you give them your public key at the moment. And if you have some private channels, you also tell them about your private channels, so they know how to route to you along those private channels.
Those two things, I think, are fairly well-known by now. Christian has a great concept, called Rendezvous Routing, which I think would help to address some of these issues, because you then don't need to give your public key or your private channels; you instead just give a partial onion to the person who's paying you and say, "Hey, this is the last bit of the route", and they don't need to know exactly where they're going to. So, that's an important one to mention.
I think what's nice about Lightning is, when you think about privacy and security on the whole, there are lots of things to be done. So, I mean, I'll maybe play Peter for a second and ask Christian a question, but in a world where we have all the things we want, what are we worried about; like, in a magical world where Taproot and Schnorr have activated perfectly, we've upgraded the Lightning Network, we have rendezvous routing, we have PTLCs, which we didn't even touch on in this talk; my question is, now what are we worried about, after we've had a huge party, we've popped the Champagne, we've all gotten over our seven-day hangover?
Because, the one thing I would say is still the on-chain concept, we do need to be thinking about. I think people could think about it a bit more, the way we handle those UTXOs, because most implementations just use every random UTXO as it comes to them. But what would you say, in the ideal world, when this is all happened, what are we still actually concerned about?
Christian Decker: I think that we'll come up with some interesting questions! No, this is a very bubbly space and so, I think both on the attacker side, or the adversarial thinking side, as well as the implementary side, we will always have a longer list of stuff that we would like to do. For example, we are now talking about Route Blinding, instead of rendezvous routing; we're talking about maybe eventually having L2; maybe doing some stuff with channel factories. So, I don't think we'll ever be done.
I'm not saying that because I think that's a sad thought, but because that's what excites everybody. We're all geeks in the end and I think this is what we end up discussing and what we end up thinking about, and we will not run out of ideas to work on. To the outside, that might look like this is all doom and gloom and everybody's under attack and nobody's safe; but we are, our inquisitive mindsets, are basically looking into how we ourselves would break this and how we could defend against this.
So, I think it's a very positive signal when there's a new paper coming out that says, "We found this new attack!" For me, that means it's time to dig in. And, for people seeing this paper, it might mean, "Oh, we're all doomed!" But the issue is, we're doomed when there's no paper, when suddenly things start breaking, when suddenly there is something that is not working and we can't explain it. So, that's when I'd be worried, not when there is a new attack being talked about; because attacks are just open questions, open challenges, for us to come up with solutions.
Peter McCormack: And I think for me, Carla, I look forward to the show where I come back and I say to you, "Talk to me about privacy on the Lightning Network", and you say, "Oh, no, it's all done. Everything is perfectly private. I mean, there's all this technical stuff in the background, but you don't have to think about anything; everything's perfectly private". And I'm like, "Okay", we record a two-minute show and we're done!
How much do you drink, by the way, for a seven-day hangover?!
Carla Kirk-Cohen: No, I'm just joking!
Peter McCormack: That's pretty brutal!
Carla Kirk-Cohen: As I get older, they get longer, so I'm assume it's just linear; they just keep going, right? I'm only on about two-day ones now!
Peter McCormack: Once you start getting to the three-day ones, you start to drink a lot less, and then you start to think about other issues, hey, Christian? Christian, are you there, or are you hungover?
Christian Decker: Yeah, I'm totally hungover on water from a Guinness glass!
Peter McCormack: I'm just sunburnt. All right, listen, I do have a closing question for you, Carla. By the way, you've been absolutely brilliant. I don't know how many podcasts you've done since I've seen you, but you've absolutely nailed it; Christian's faith in you was fully deserved. But do you just want to talk about what you're working on right now; do you want to tell people what you're doing?
Carla Kirk-Cohen: Sure, I'll keep it short. I work on Lightning Loop, so if anyone's experiencing any issues or has any feedback on that, please do publish it on our GitHub. Lightning Labs is hiring, got to throw that one in. If you are an organisation who are looking to support free and open-source development, please check out Brink. And other than that, my Twitter DMs are open; nothing bad has happened so far, so if anything was unclear or anyone's interested in getting stuck in on that, hit me up on Twitter.
Peter McCormack: What is your Twitter?
Carla Kirk-Cohen: It's @actuallyCarlaKC, because CarlaKC is a scammer, so don't follow her!
Peter McCormack: Okay, I'll put that in the show notes. Do you know what I also think we should do? I think we should get you, Amiti and, was it Fabian; was that his name?
Carla Kirk-Cohen: Yes. There's a reunion; we've been dying for one, we did all calls during lockdown and all of that. So, Resi reunion would be amazing.
Peter McCormack: Yeah, Resi reunion. We'll get, God, why have I lost his name for a second? It's going to be so embarrassing. Oh, Newbery; get John Newbery back with us and the three of you. We'll do a reunion, talk about it, and try and encourage people to make donations towards open-source and development, because I think that's obviously a permanent and important topic that I'm actually talking to John Pfeiffer about later today as well.
So, brilliant, lovely to see you again, Carla. I'm glad to see you're absolutely crushing it and you must be very proud, Christian, and good to see you again. Christian, tell people where they can follow you?
Christian Decker: Yeah, I'm on Twitter @Snyke and CDecker almost everywhere else! So, reach out to me if you have any questions or if you're interested in just talking Lightning, I always enjoy talking tech.
Peter McCormack: Well, keep up the amazing work. Honestly, for people like me who are quite challenged on the technical side, a little bit lazy as well in my understanding, just to go out to somewhere like El Salvador and to be able to use the network every day perfectly, I've never had a single failed payment when I've been there, it's down to the great work of people like you and I get to live on the shoulder of what I think are technical giants. So, I think you deserve a lot of credit for the amazing work you to do contribute to the network.
Thank you both for coming on and talking me through this. Keep doing it, keep crushing it and hopefully, see you soon.
Christian Decker: Will do.
Carla Kirk-Cohen: Cheers, Peter.
