Latitude and longitude are defined "relative to" a point off the coast of Ghana (0 degrees lat., 0 degrees long.). There is nothing special about that point; it is arbitrary. You don't "have to start there before you even get directions in the first place." That is nonsense. Directions are independent of our arbitrary definitions of latitude and longitude; they are defined based on physical phenomena such as the path of the sun, or the direction of earth's magnetic field. How do you think people found their way anywhere before the random location off the coast of Ghana was ever agreed upon as the reference point or before mankind had any conception of latitude and longitude?

Any point on the earth can be used as a reference point. If I "am" the U.S. (i.e., if I consider any reference point in the U.S.), my most easterly point is in Maine because no matter where I start from in the U.S., going in an eastward direction, Maine is the furthest point I can get to before having traveled halfway around the world. (To get anywhere beyond halfway around the world, I would have started off heading west, not east.) Yes, Semisopochnoi has the most eastern longitude, but that is effectively simply the answer to the following trivia question: "If you start just off the coast of Ghana and head east, what is the furthest location in the U.S. you can reach before having traveled exactly halfway around the earth." For the same reason as Maine is the easternmost point of the U.S., Attu Island--NOT Amatignak Island--is the westernmost location of the U.S. even though Amatignak Island has the most western longitude.

And once again, I did not argue that directions change; I argued that the description of where a given place is changes depending on the reference point. Wherever that reference point happens to be, everything in the hemisphere in the westerly direction from that reference point is "west" of that point, while everything in the hemisphere in the easterly direction from that reference point is "east" of that point.