Why Self-Driving Cars Must Be Programmed to Kill
Pages: 12
closed account (z05DSL3A)
"Self-driving cars are already cruising the streets. But before they can become widespread, carmakers must solve an impossible ethical dilemma of algorithmic morality..."
http://www.technologyreview.com/view/542626/why-self-driving-cars-must-be-programmed-to-kill/
http://www.technologyreview.com/view/542626/why-self-driving-cars-must-be-programmed-to-kill/
Part of what is not being addressed is the safety of the infrastructure over which cars (and pedestrians, for example) operate.
I always laugh at films like Star Wars or The Fifth Element that show all these flying cars zooming around -- driven by people, even if most of the non-computerized drivers are Jedi -- without a catastrophic death toll.
The danger of cars was clearly laid out in the 60s, but people want their convenience more than they want safety.
And therein lies the problem, methinks.
Compounded by moralizing which innocent is more likely to die, based upon money.
I always laugh at films like Star Wars or The Fifth Element that show all these flying cars zooming around -- driven by people, even if most of the non-computerized drivers are Jedi -- without a catastrophic death toll.
The danger of cars was clearly laid out in the 60s, but people want their convenience more than they want safety.
And therein lies the problem, methinks.
Compounded by moralizing which innocent is more likely to die, based upon money.
Why choose to kill one or the other? Why not just slightly injure both by braking really quickly? In that case nobody dies.
Right at the start of the article it says it is talking about unavoidable accidents. Such as maybe a car pulls out right in front of the self-driving car with not enough room for the self-driving car to stop itself or slow down enough that the collision would not be fatal. Should it swerve onto the pavement and kill the people on the pavement if it can, or should it instead choose to kill the driver rather than risk killing many more? It's the algorithm designers' version of the Kobayashi Maru.
Last edited on
If it is possible for the car to be in such a situation, it must have been intentionally designed poorly.
If there isn't enough information to proceed safely, it should proceed as slowly as reasonable. That means if driving faster could ever lead to such an ethical dilemma, then it should just drive slower. Problem solved. In most cases the car would still move at a normal speed.
EDIT: Computergeek01 +1
If there isn't enough information to proceed safely, it should proceed as slowly as reasonable. That means if driving faster could ever lead to such an ethical dilemma, then it should just drive slower. Problem solved. In most cases the car would still move at a normal speed.
EDIT: Computergeek01 +1
Last edited on
Wow, that's some well put together sensationalism. I mean which piece do we pick apart first? The idea that crowds of people randomly walk out in front of speeding cars on a daily basis? Or should we address the fact that since there is no need to navigate the vehicle, the occupants inside of it will likely be situated so that they are facing the back which drastically increases the survivability rate in car accidents (think of car seats for little kids) and that you can employ much more effective breaks than what we have now because in this configuration you don't have to worry about whiplash? How about my personal favorite idea that this is actually an infrastructure problem and pedestrian bridges would not only solve this issue but they would also increase the throughput for both motorists and people on foot? Then again, that one should have been done 20 years ago so we can probably stop holding our breath and waiting for common sense to take hold.
The real problem with safety and self driving cars will be with the modern road hazards that the municipalities out right refuse to properly deal with. I'm talking about the bicyclists, skateboarders, hover-borders, joggers etc. that can't actually keep up with the flow of traffic, constantly break established traffic laws and still for some reason insist that they have a right to be there. Nowadays when one of these jack-asses gets hurt while doing something impossibly dumb, like lane-splitting in an intersection during rush-hour, people blame the driver. But when you remove that scapegoat the blame is going to land on the self driving cars. I should also mention that those people who don't buy into the self-driving car idea will likely become new road hazards themselves.
EDIT: *slow clap* for shadowmouse, you definitely win King of the Nerds with that Kobayashi Maru reference. I didn't even know what you were talking about until I read the first sentence in Wikipedia and it all hit me again like a wave.
In the spirit of the game though, the people in the street are the ones that put themselves in harms way; why should anyone else be put at risk due to their collective failure in situational awareness? How many times were you told as a kid to look both ways before crossing the street? If they can't be bothered to follow basic traffic safety then there is ultimately little that we can do to save them from themselves.
The real problem with safety and self driving cars will be with the modern road hazards that the municipalities out right refuse to properly deal with. I'm talking about the bicyclists, skateboarders, hover-borders, joggers etc. that can't actually keep up with the flow of traffic, constantly break established traffic laws and still for some reason insist that they have a right to be there. Nowadays when one of these jack-asses gets hurt while doing something impossibly dumb, like lane-splitting in an intersection during rush-hour, people blame the driver. But when you remove that scapegoat the blame is going to land on the self driving cars. I should also mention that those people who don't buy into the self-driving car idea will likely become new road hazards themselves.
EDIT: *slow clap* for shadowmouse, you definitely win King of the Nerds with that Kobayashi Maru reference. I didn't even know what you were talking about until I read the first sentence in Wikipedia and it all hit me again like a wave.
In the spirit of the game though, the people in the street are the ones that put themselves in harms way; why should anyone else be put at risk due to their collective failure in situational awareness? How many times were you told as a kid to look both ways before crossing the street? If they can't be bothered to follow basic traffic safety then there is ultimately little that we can do to save them from themselves.
Last edited on
closed account (z05DSL3A)
It is a hypothetical scenario, get over it and look at what they are investigating.
I didn't mean any hostility toward you Grey Wolf, I simply wanted to point out that this is clickbait. I think the topic of how these devices should behave would be a great topic of discussion. I was just disappointed at the way this article is framing it. Better questions to ask would be things like how should these vehicles behave in an emergency like driving someone to the hospital? Or a non-life threatening emergency like a father missing the birth of their child or somebody running late for a plane or a job interview? Should there be some kind of override for the devices normal behavior? The cars are safer and less likely to make mistakes, but how do we prevent abuse of such a feature? If you remove the risk of personal injury what is to stop someone from running around all day in "Emergency Mode"? These are the questions I would like to see discussed on this topic.
EDIT: More questions: Should it be a criminal act to "tweak" the car's behavior for the drivers benefit? How do they deal with signal loss for the purpose of navigation?
EDIT: More questions: Should it be a criminal act to "tweak" the car's behavior for the drivers benefit? How do they deal with signal loss for the purpose of navigation?
Last edited on
Again, I agree that infrastructure is the prime issue, but to be fair, we must resign ourselves to what is feasible for the car manufacturer:
- he cannot require municipalities to redesign extant roads and byways
- he cannot eliminate dangerous roads or intersections
- he cannot change people's dangerous behaviors
What he can do is try to minimize the impact of a bad, extant situation.
Go to any university. IDK about foreign countries, but here in the states (specifically, NJ), a driver is required to yield to pedestrians in a crosswalk.
Unfortunately, many students take that to mean 'if I am struck in a crosswalk it is the driver's fault', and carelessly step out into oncoming traffic, simply because his foot falls in a marked crosswalk. People who drive in those locations quickly learn that idiots will jump off the curb into the path of their moving vehicles and drive with that in mind, in order to avoid hitting them.
Being in a car that strikes something is significantly less dangerous than being struck by a car. Car manufacturers have spent zillions of dollars designing cars to distribute an impact on the occupants over a longer time, because that's where the data shows people are hurt most.
I had a friend who was driving home with a truck he had just purchased, actually travelling under the 30mph speed limit, in an inner city street, green lights, with traffic, when someone in another vehicle shot out from a blind side street so fast, and so suddenly, that the collision was unavoidable. A computer may have been able to react fast enough to have prevented his new truck from being totaled, but there lies the limits to simplistic, non-integrated systems.
I personally was driving, at about 20mph, through Camden, in the middle of a string of traffic, when a woman, who had been standing placidly on the curb, without looking simply stepped out in front of my car. Had I not been wary that she might do something like that (I am what most people consider to be an excessively careful driver), I would have hit her. And she was angry, until I pointed out that it was she who stepped into moving traffic. (She was rattled after that.) I am fortunate that the person behind me didn't hit me!
The Schuylkill Expressway is known as a dangerous drive -- it has a great big curve, with high walls -- high speed, no forward vision -- and there are constantly auto collisions. Yet people still speed around those blind curves, every minute of every day.
I don't believe in 'accidents', you see, as the only accident is the same as 'accidentally falling on the scissors you are running around with'.
You can't change things like this without a change in the way people think about and behave in and around automobiles. And the data shows that to be overwhelmingly improbable -- at least without pumping the kind of money into it that no one can justify spending. The rates of death on our roads should have long ago caused people to curse cars and find a better way.
But they're so nice and convenient, you see.
It's the person who drives hard in traffic, speeding, swerving around other motorists -- who invariably causes an auto collision two miles down the highway.
It's the person speeding around the mountain bends late at night, who you pass five miles down the road with a deer embedded in their grill.
It really is a Kobayashi Maru scenario. And we are the Klingons.
So when the little kid following the bouncing ball jumps out from behind a parked car into your path, does your vehicle swerve into the schoolbus passing in the opposite direction, or do you skid into the little kid? The little kid might die, but you won't be facing lawsuits from thirty parents and the school district and be out a wrecked car. You might be obligated to attend a funeral.
Cold? Sick?
You bet.
Welcome to the way things are. Claiming that you can simply change that is fantasy. That kind of change must be affected another way, with long-term, oblique goals. Which include eliminating the possibility that a child can surprise you from behind a vehicle parked on the side of the road. (And yes, I've had kids deliberately jump out in front of my car, because they were too young and stupid to know better!)
I think that satellite tracking is part of the answer... to identify hazards before you (or your autonomous vehicle) can perceive it.
Bleh.
- he cannot require municipalities to redesign extant roads and byways
- he cannot eliminate dangerous roads or intersections
- he cannot change people's dangerous behaviors
What he can do is try to minimize the impact of a bad, extant situation.
Go to any university. IDK about foreign countries, but here in the states (specifically, NJ), a driver is required to yield to pedestrians in a crosswalk.
Unfortunately, many students take that to mean 'if I am struck in a crosswalk it is the driver's fault', and carelessly step out into oncoming traffic, simply because his foot falls in a marked crosswalk. People who drive in those locations quickly learn that idiots will jump off the curb into the path of their moving vehicles and drive with that in mind, in order to avoid hitting them.
Being in a car that strikes something is significantly less dangerous than being struck by a car. Car manufacturers have spent zillions of dollars designing cars to distribute an impact on the occupants over a longer time, because that's where the data shows people are hurt most.
I had a friend who was driving home with a truck he had just purchased, actually travelling under the 30mph speed limit, in an inner city street, green lights, with traffic, when someone in another vehicle shot out from a blind side street so fast, and so suddenly, that the collision was unavoidable. A computer may have been able to react fast enough to have prevented his new truck from being totaled, but there lies the limits to simplistic, non-integrated systems.
I personally was driving, at about 20mph, through Camden, in the middle of a string of traffic, when a woman, who had been standing placidly on the curb, without looking simply stepped out in front of my car. Had I not been wary that she might do something like that (I am what most people consider to be an excessively careful driver), I would have hit her. And she was angry, until I pointed out that it was she who stepped into moving traffic. (She was rattled after that.) I am fortunate that the person behind me didn't hit me!
The Schuylkill Expressway is known as a dangerous drive -- it has a great big curve, with high walls -- high speed, no forward vision -- and there are constantly auto collisions. Yet people still speed around those blind curves, every minute of every day.
I don't believe in 'accidents', you see, as the only accident is the same as 'accidentally falling on the scissors you are running around with'.
You can't change things like this without a change in the way people think about and behave in and around automobiles. And the data shows that to be overwhelmingly improbable -- at least without pumping the kind of money into it that no one can justify spending. The rates of death on our roads should have long ago caused people to curse cars and find a better way.
But they're so nice and convenient, you see.
It's the person who drives hard in traffic, speeding, swerving around other motorists -- who invariably causes an auto collision two miles down the highway.
It's the person speeding around the mountain bends late at night, who you pass five miles down the road with a deer embedded in their grill.
It really is a Kobayashi Maru scenario. And we are the Klingons.
So when the little kid following the bouncing ball jumps out from behind a parked car into your path, does your vehicle swerve into the schoolbus passing in the opposite direction, or do you skid into the little kid? The little kid might die, but you won't be facing lawsuits from thirty parents and the school district and be out a wrecked car. You might be obligated to attend a funeral.
Cold? Sick?
You bet.
Welcome to the way things are. Claiming that you can simply change that is fantasy. That kind of change must be affected another way, with long-term, oblique goals. Which include eliminating the possibility that a child can surprise you from behind a vehicle parked on the side of the road. (And yes, I've had kids deliberately jump out in front of my car, because they were too young and stupid to know better!)
I think that satellite tracking is part of the answer... to identify hazards before you (or your autonomous vehicle) can perceive it.
Bleh.
I've never gotten hurt while lane-splitting through fast traffic, but I've been rear-ended while going in a straight line through a preferential lane, bumped to the ground by a close pass on my left (also while moving in a straight line), and side-swiped from the right by an unsignaled left turn by a slower vehicle. Let's not go into the almost weekly high-speed cut-offs, or my nascent precognitive skills that arise from a need to survive the near-constant stream of cars that don't signal their maneuvers ahead of time.
Should I follow a law not intended for my type of vehicle even at the cost of my own safety?
That "some reason" is an actual right in most states of the US and in most countries in the world.
| constantly break established traffic laws |
| still for some reason insist that they have a right to be there |
I don't see why most of you are insisting this article should have been a different article. Of course these cars have to be programmed to handle these life or death situations, even if the events are unlikely, or would be rare. You can't assume nothing will go wrong, even if you have redesigned a roadway or whatnot. This is a real morality issue that must be decided. This article is about that issue, not about other issues like can the user override the vehicle, or can the vehicle be programmed to have a zero probability of collision, or can the roadways be designed so that nobody ever gets hurt.
Last edited on
Honest question Duoas: do you really want to have everyone's movements tracked by satellite? I don't know how much you've seen about misuses of the RIPA but some pretty serious privacy breaches have been carried out with a lot less than that. And besides, who would actually make and launch the worldwide tracking satellites? Unless of course you mean using currently existing ones in which I don't know about you but I'd rather not have my exact position known at all times.
Also, please clarify what you meant about the Kobayashi Maru. What I was saying is that the algorithm designer must choose between a situation which involves killing one group of people (in this case pedestrians or in the test, leaving the crew of the KM to die), or killing oneself (in this case the driver or in the test entering the neutral zone so as to save the KM and being killed while doing so). This is almost the exact situation of the test from the point of view of the person who is taking it.
Finally, @ComputerGeek01 thank you for the applause and coronation, I'll remember that, it means a lot to me.
Also, please clarify what you meant about the Kobayashi Maru. What I was saying is that the algorithm designer must choose between a situation which involves killing one group of people (in this case pedestrians or in the test, leaving the crew of the KM to die), or killing oneself (in this case the driver or in the test entering the neutral zone so as to save the KM and being killed while doing so). This is almost the exact situation of the test from the point of view of the person who is taking it.
Finally, @ComputerGeek01 thank you for the applause and coronation, I'll remember that, it means a lot to me.
@ helios: It's not that I am completely against sharing the road with bicyclists. Here in the States there is an organization called ABATE; when I was growing up the guy my mother was dating hung around with that crowd and the difficulties you are describing are almost parallel to their own. But it's a public space and a certain amount of cooperation is needed. Not passing a drive through their blind-spot from a direction they weren't expecting is part of that cooperation. I'll never understand the need for lane-splitting, you are already aware that visibility is an issue for you yet you consciously add to the problem. Why? And what laws would compromise your safety? If you're in danger of being run over then it's reasonable to expect that you'd get out of the way. Otherwise signaling before you change lanes is as much for your safety as everyone else.
Up here we have these idiots that ride around poorly lit streets all night with dark clothing and no helmet. I don't mean streets where a light happened to go out. I mean places that have been that way for 50+ years. Literally the only hint you have that they are there is that tiny reflector on the back of the bike and trust me you have to be looking for it to see it. They even wear baggy clothes so that you don't even have a human shaped silhouette to go by. Then they act indignant when one of them gets hit. These are the people that I have an issue with. These are the people who I will say are too stupid to share the same road as everyone else.
Up here we have these idiots that ride around poorly lit streets all night with dark clothing and no helmet. I don't mean streets where a light happened to go out. I mean places that have been that way for 50+ years. Literally the only hint you have that they are there is that tiny reflector on the back of the bike and trust me you have to be looking for it to see it. They even wear baggy clothes so that you don't even have a human shaped silhouette to go by. Then they act indignant when one of them gets hit. These are the people that I have an issue with. These are the people who I will say are too stupid to share the same road as everyone else.
| shadowmouse wrote: |
|---|
| do you really want to have everyone's movements tracked by satellite? |
| I don't see why most of you are insisting this article should have been a different article. |
We have all, however, engaged in a conversation on points relating to the article, in an off-topic forum of an online C++ site. Woah.
| shadowmouse wrote: |
|---|
| Honest question Duoas: do you really want to have everyone's movements tracked by satellite? |
With our current technology and accountability and security? Of course not.
But it's happening already.
And we're paying our service providers to do it.
| ibid wrote: |
|---|
| Also, please clarify what you meant about the Kobayashi Maru. |
Well, I was using only the most general consideration -- in the KM, anyone testing who chooses for anyone to die has failed -- the KM is a "no-win scenario". No matter what you do, the KM and all its crew will die. The point of the KM is to see how the crew will react under the pressures of a no-win, we're-all-gonna-die scenario.
As relates to our horrible driving scenarios, the crew has already crossed into the Neutral Zone. Yours is, as in the article, whom do I save first: myself or others?
Mine is, we've caused the problem to begin with. By running around with knife in our hands, we've invited ourselves to have to decide who gets stabbed as we're rolling down the stairs. The problem could have been avoided by sheathing the knife and walking. We are the Klingons. We've placed ourselves in the position of having to make life and death decisions.
For example, we've designed our neighborhoods such that our houses open to streets through which two-or-more-ton boxes are regularly motored with incredible speeds by distracted people. And our kids go out and play in those self-same streets.
And then it's such a terrible, tragic, accident when something goes wrong.
So, let's all move somewhere else for a year, and pay the expense, of having our houses razed and replaced with a superior layout and design. Right?
The economics of it all make us the Klingons as much as the poor saps who are going to die for the Kobayashi Maru.
| LB wrote: |
|---|
| If you want it to be private, don't do it outside where anyone can see it. |
I agree, but unfortunately the law does not.
While I agree that ninja-cycling is very reckless, I don't agree that the main danger to cyclists comes from a visibility problem. If a driver exercises due care, nearby cyclists can easily anticipate their movement, regardless of whether that driver can see them. If a driver is being careless, even a Christmas tree on a bike will be in danger.
I feel safest while navigating dense, flowing traffic because cars have few maneuvering options and any maneuvers they do make, they make slowly and carefully, giving me ample time to plan ahead. There isn't much difference in risk levels when filtering vs. when riding like a car. While filtering I'm slightly harder to see and have fewer exit strategies, but I'm not in as much danger in case of a sudden stop.
Actually, one of the most dangerous situations, which I've learned to recognize after a couple close calls, are a long queue of waiting cars next to a long stretch of empty lane, since cars often bolt out very quickly when the driver loses their patience.
I think cyclists are in most danger from two separate phenomena: a) many drivers are not attentive enough and/or mistakenly believe that their awareness is complete, leading them to perform dangerous actions under the sometimes incorrect assumption that they're safe to perform at that time. The causes for this are many (tiredness, stress, Dunning-Kruger, DUI, personality disorders, etc.). And b) drivers sometimes dangerously bend the law for convenience's sake; the most common examples I see of this are failure to indicate and improper turns (in one particular instance I saw a car turn right across all four lanes of the road).
Some of these are intrinsic aspects of the human condition, and the only way to really solve them is by taking the human element out.
I see no reason why automatic cars would increase the risk for non-motor road users; if anything I think it'll substantially decrease it by having more predictable drivers. Yes, there are reckless cyclists; I'm saying that automatic cars pose no greater danger to them than human drivers.
EDIT: I forgot these two:
Most notably, traffic lights, which force me to share space with cars when I'm at maximum instability. I love those "red lights are yield signals for cyclists" laws, and really wish they'd implement that here.
Although it doesn't quite fit my earlier statement, laws that force cyclists to the absolute edge of the road are complete BS, since they ignore that some cyclists can actually move as fast as motor traffic under certain conditions, and they leave the cyclists that follow it no exit strategies in case they need to avoid some hazard.
Why? Are cars a force of nature, rather than machines controlled by sentient beings?
I feel safest while navigating dense, flowing traffic because cars have few maneuvering options and any maneuvers they do make, they make slowly and carefully, giving me ample time to plan ahead. There isn't much difference in risk levels when filtering vs. when riding like a car. While filtering I'm slightly harder to see and have fewer exit strategies, but I'm not in as much danger in case of a sudden stop.
Actually, one of the most dangerous situations, which I've learned to recognize after a couple close calls, are a long queue of waiting cars next to a long stretch of empty lane, since cars often bolt out very quickly when the driver loses their patience.
I think cyclists are in most danger from two separate phenomena: a) many drivers are not attentive enough and/or mistakenly believe that their awareness is complete, leading them to perform dangerous actions under the sometimes incorrect assumption that they're safe to perform at that time. The causes for this are many (tiredness, stress, Dunning-Kruger, DUI, personality disorders, etc.). And b) drivers sometimes dangerously bend the law for convenience's sake; the most common examples I see of this are failure to indicate and improper turns (in one particular instance I saw a car turn right across all four lanes of the road).
Some of these are intrinsic aspects of the human condition, and the only way to really solve them is by taking the human element out.
I see no reason why automatic cars would increase the risk for non-motor road users; if anything I think it'll substantially decrease it by having more predictable drivers. Yes, there are reckless cyclists; I'm saying that automatic cars pose no greater danger to them than human drivers.
EDIT: I forgot these two:
| And what laws would compromise your safety? |
Although it doesn't quite fit my earlier statement, laws that force cyclists to the absolute edge of the road are complete BS, since they ignore that some cyclists can actually move as fast as motor traffic under certain conditions, and they leave the cyclists that follow it no exit strategies in case they need to avoid some hazard.
| If you're in danger of being run over then it's reasonable to expect that you'd get out of the way. |
Last edited on
I've been riding a bike more than half of my life and continue riding to work, but as of a year ago, I am also a car owner.
As a bike rider, I am a ninja. I never hand signal, I ride on sidewalks, and in general don't care about trafic lights if there are no cars around.
However, on intersections, I always make eye contact with the drivers. I make sure they see me if I am to cross their way. When I do ride on sidewalks, that's because the lane is narrow or there are parked cars.
As a car driver, I am the most careful driver among my friends and wife. When there are bikers around, I always remember to pay attention for bikes, especially if they are slaloming between cars stopped for trafic lights.
***
Self driving cars would be awesome. Yesterday I was drinking beer with friends; my wife is the designated driver so no problem for me, but one of my friends was on his own, and was drinking just as much as me. I asked him: on how many beers do you drive your car? He said, on 6-7 beers, I still drive. I asked, what if the cops catch you? He said he drives well, so no worries for him. The guy is way too macho, and stupidly so. I should have asked him: what if you kill someone while driving drunk? Will you dare say it was unavoidable and that alcohol didn't play a role?
Self driving cars are very very needed.
As a bike rider, I am a ninja. I never hand signal, I ride on sidewalks, and in general don't care about trafic lights if there are no cars around.
However, on intersections, I always make eye contact with the drivers. I make sure they see me if I am to cross their way. When I do ride on sidewalks, that's because the lane is narrow or there are parked cars.
As a car driver, I am the most careful driver among my friends and wife. When there are bikers around, I always remember to pay attention for bikes, especially if they are slaloming between cars stopped for trafic lights.
***
Self driving cars would be awesome. Yesterday I was drinking beer with friends; my wife is the designated driver so no problem for me, but one of my friends was on his own, and was drinking just as much as me. I asked him: on how many beers do you drive your car? He said, on 6-7 beers, I still drive. I asked, what if the cops catch you? He said he drives well, so no worries for him. The guy is way too macho, and stupidly so. I should have asked him: what if you kill someone while driving drunk? Will you dare say it was unavoidable and that alcohol didn't play a role?
Self driving cars are very very needed.
Last edited on
So far all of the accidents involving self-driving cars were the fault of the other car. They're just safer all around. They don't get sleepy, they don't get drunk, they don't get texts or calls, and they don't eat food while driving.
Last edited on
closed account (48T7M4Gy)
http://next.mercedes-benz.com/en/autonomous-truck-on-german-roads/
http://next.mercedes-benz.com/en/new-mobility-in-san-francisco/
http://next.mercedes-benz.com/en/new-mobility-in-san-francisco/
Last edited on
I just had a vision where your car not only drives itself, it stops and refuels itself, and it also knows when it needs maintenance and drives itself into an autonomous-auto repair shop. It also washes and vacuums itself when it gets too dirty and has itself touched up when someone scratches it's paint. The future is going to be awesome.
Pages: 12