TLS is application layer, on top of TCP. It's not part of TCP itself.
Sockets libraries (both on Unix and Windows) allow opening raw sockets at the transport layer. You can assemble your own TCP packets (or use some other transport protocol). Obviously you'll have to implement TCP entirely yourself.
Raw sockets can also be opened at the data link (e.g. Ethernet) and network (e.g. IP) layers.
Because when I send a https request from the browser to my server, the message is encrypted (I receive 517 bytes, but the output is 3 strange characters)
If this is the initial part of the handshake then the message shouldn't be encrypted yet?
Sounds like you're trying to do https yourself. You can, but it's rarely done because it's complicated, tricky to get right, and you have to rewrite it all when a new protocol is used (currently the world typically uses TLS 1.2).
There are many libraries for C, but we tend to use openssl, not because it has a nice interface, but because it works.
There's an initial Client Hello, the server replies with it's own Hello and sends the cyphers it accepts. The client chooses one and they do a key echange (using expensive public keys to generate one time fast cheap key).
Then they talk. It's hard to get right by yourself.
C++ is the fastest thing around, but by and large, the libraries are non-existent or suck.
As openssl is such a pain to use directly, it's often easier to to this stuff using something like Poco, which wraps openssl in a more application-centric interface.
On windows I only got 3 characters. This is the output on linux, maybe it tells u something:
it tells me that you are trying to print binary data onto the screen.
An unsigned byte has 0-255 (256, 8 bits worth) total values. When printing to the screen a subset of those are printable (many of the ones near zero are not printable) and you use the ones that make sense for text. In binary any byte can be any value and trying to print it gives nonsense. The only sensible way to print binary to the screen is 1 byte at a time, usually in hex but integer format is ok at times, so you can see the true value rather than gibberish.
in windows, this is on par with trying run this at the console and expecting something useful:
c:\ type a.exe
flush and close the file. If that does not work, try writing hello world or something to the file in hex. Debug it from the bottom up, in other words... get the file working, once you can see something there, try the packet data again, if that does not work, make sure you rx'ed a packet at all, etc...
what he gave is what I was saying to do. or in c++, I think its just
cout << hex << buffer[index] << " "; //or filename instead of cout as needed
wireshark is your friend, but, as you noted, you do need to see what you have in hand yourself, so you can begin to unravel it.