Passing Data Over Network (Serialize ?)

[Kartoffel]

Hello,

I'm currently developing a server which must transfer much data in a short amount of time. This kind of data is mainly some command such as 'CMD*PARAM*PARAM*PARAM', how would one be able to reduce the size of this ?

I've heard about Serilization & Binarization, how would one proceed to accomplish this knowing it must be portable between Windows & Linux ?

I've been doing some researchs and messed around with Boost, however I did not seem to get it working as expected.

Thanks for your help.

[helios]

Serialization doesn't reduce the size of data. What serialization does is transform structured data, such as trees and graphs, into a serial stream. From what you're saying, your data is already a serial stream, so there's nothing else to do with it.

If the connection between the client and the server is slower than ~10 MiB/s, or 100 Mbit/s, compressing the stream with a fast algorithm like LZ4 or gzip at level 4-5 should compress to 40% of the original size without the compression becoming a bottleneck. That would increase the throughput to around 250 Mbit/s. Do not use more algorithms with higher compression ratios such as bzip2 and LZMA, because they're slower and you'll likely end up reducing the throughput.
If the connection is considerably slower than 100 Mbit/s, using these algorithms may make sense.
If the connection is on the order of a gigabit or faster, you should not attempt any compression at all.

[Kartoffel]

Sorry for the late reply.

How would I transfer things then ? Shall I just do it in plain text ? Analyzing packets of other servers doesn't show up much of plain text, do they do something else ?
Is there no other way ? I got told that putting in text was way more weight costly than putting it in binary.

[jlb]

What kind of server?

What protocols are you planing on using?

Are you sure you need to be so worried about the transfer speed?

I myself would start with plain text, then once you have that working you can think about other methods. Plain text is much easier to see what is going wrong and once you have a working connection you can start profiling different sections of the code to see where the bottleneck, if any, exists.

[Kartoffel]

Basically the server would have to transfer commands with their parameters in a short amount of time.
UDP is used to achieve this, it's for a game, it must synchronize 3D players between themselves.
Currently I am using plain, and effectively, much more better for debug, however I'm planning ahead.
It coiuld works in plain, however, I don't want it just to work, I'd like it to be optimized too.

[jlb]

If it works, it works. How do you know it isn't optimized? Have you profiled the code to see if there is an issue? Optimizing without knowing that there is a problem and where the problem is located can make the imagined problem worse.

[Kartoffel]

No, what I meant by optimizing, is to use the least resources possible, even though there are not much used.
Example, why using 50KB/s while we can use 25. Anyone can handle a 50KB/s, but it'd always be for the better (unless drastically changing the message) to have its size halved.

[helios]

What kind of data will you move between the computers?
Keep in mind that anything you do to increase the bandwidth will also increase the latency, and low latency is much more important for games than high bandwidth and throughput.

[Kartoffel]

Yup', I effectively know.

What I'd need to transfers are positions of a lot of different game objects.
And yup', I know, that's the reason why I wanted to lower the size of each messages.

[helios]

Then the fastest way to encode that information is in a binary stream.
For example,

template <typename DstT, typename SrcT>
void serialize_fixed_le_int(DstT *dst, const SrcT &src){
	static_assert(CHAR_BIT == 8, "Only 8-bit byte platforms supported!");

	auto p = (std::uint8_t *)dst;
	typename std::make_unsigned<SrcT>::type copy = src;
	const auto n = sizeof(src);
	for (auto i = 0; i != n; i++){
		p[i] = copy & 0xFF;
		copy >>= 8;
	}
}