This example you show us prints a C-string. So, to print whole data, you must know the length of it and use a loop similar to the one you use for reading it. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams?
Collectives on Stack Overflow. Learn more. Asked 7 years, 6 months ago. Active 8 months ago. Viewed 35k times. Improve this question. Is your problem about doing something with the read data? Because the reading seems ok assuming open mode is std::binary , and the use of the buffer as well.
Add a comment. Active Oldest Votes. The bytes in [brackets] are the UTF8 of "testing". The key here is 0x It's parsed:. And here's the encoded version, again with the Test1's a field set to These repeated values do not have to appear consecutively; they may be interleaved with other fields.
The order of the elements with respect to each other is preserved when parsing, though the ordering with respect to other fields is lost. In proto3, repeated fields use packed encoding , which you can read about below. For any non-repeated fields in proto3, or optional fields in proto2, the encoded message may or may not have a key-value pair with that field number.
Normally, an encoded message would never have more than one instance of a non-repeated field. However, parsers are expected to handle the case in which they do.
For numeric types and strings, if the same field appears multiple times, the parser accepts the last value it sees. For embedded message fields, the parser merges multiple instances of the same field, as if with the Message::MergeFrom method — that is, all singular scalar fields in the latter instance replace those in the former, singular embedded messages are merged, and repeated fields are concatenated. The effect of these rules is that parsing the concatenation of two encoded messages produces exactly the same result as if you had parsed the two messages separately and merged the resulting objects.
That is, this:. This property is occasionally useful, as it allows you to merge two messages even if you do not know their types. Version 2. In proto3, repeated fields of scalar numeric types are packed by default. These function like repeated fields, but are encoded differently. A packed repeated field containing zero elements does not appear in the encoded message. Otherwise, all of the elements of the field are packed into a single key-value pair with wire type 2 length-delimited.
Each element is encoded the same way it would be normally, except without a key preceding it. Now let's say you construct a Test4 , providing the values 3, , and for the repeated field d. Then, the encoded form would be:. Only repeated fields of primitive numeric types types which use the varint, bit, or bit wire types can be declared "packed". The first argument to buffer. It happens to default to utf-8 so this argument is extraneous.
This means that we wrote to five bytes of the buffer. The fact that the string "Hello" is also 5 characters long is coincidental, since each character just happened to be 8 bits apiece. This is useful if you want to complete the message:. When buffer. Probably the most common way to read buffers is to use the toString method, since many buffers contain text:.
Again, the first argument is the encoding. In this case, it can be seen that not the entire buffer was used! Thus CSequentialStreamWriter should have a separate method, let's say Finalize final buffer flush , which should write to disk the last portion of data. Error handling. While the code start filling 2nd buffer, and the 1st one is being written on a separate thread, but write fails for some reason, the main thread should be aware of that failure.
Let's assume the interface of a CSequentialStreamWriter has Write function returns bool or throws an exception, thus having an error on a separate thread, you have to remember that state, so next time you call Write or Finilize on the main thread, the method will return False or will throw an exception. And it does not really matter at which point you stopped filling a buffer, even if you wrote some data ahead after the failure - most likely the file would be corrupted and useless.
I'd suggest trying file mapping. I would also advice you to look into memory map. That may be your answer. Once I had to process a 20GB file in other to store it in the database, and the file as not even opening. So the solution as to utilize moemory map. I did that in Python though. At least the MSVC implementation copies 1 char at a time to the output buffer when a stream buffer is not set see streambuf::xsputn. So one can imagine, had I a faster disk, the fstream write would slow down sooner than the stdio version.
I mean do not pass the whole "many-many-bytes" buffer at once, do a couple of writes i. OutBufferSize can be from bytes to megabyte. Keep in mind that traversing the entire N-gigabyte array also takes some time. While you are writing a small buffer, it gets to the cache and works faster. That means for most of the copy process, the CPU will basically do nothing other than telling the disk controller where to put, and get data from, eliminating a whole step in the chain, and one that is not at all optimized for moving large amounts of data - and I mean hardware.
What you do involves the CPU a lot. I want to point you to the "Some calculations to fill a[]" part. Which I think is essential. You generate a[], then you copy from a[] to an output buffer thats what fstream::write does , then you generate again, etc.
Also, when writing lots of data to files it is sometimes faster to logically extend the file size instead of physically, this is because when logically extending a file the file system does not zero the new space out before writing to it. It is also smart to logically extend the file more than you actually need to prevent lots of file extentions.
Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. Ask Question. Asked 9 years, 5 months ago. Active 1 year, 6 months ago. Viewed k times. Any ideas how I can speed up my writing? Improve this question.
0コメント