mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <>
Subject [GitHub] [mesos] bmahler commented on issue #356: libprocess: check protobuf (de)serialisation success.
Date Tue, 07 Apr 2020 19:59:41 GMT
bmahler commented on issue #356: libprocess: check protobuf (de)serialisation success.
   Took a look at the protobuf code:
   > Before the code didn't check at all the return value of
   > Message::SerializeToString, which can fail for various reasons,
   > e.g. out-of-memory, message too large, or invalid UTF-8 string.
   As far as I see, `MessageLite::SerializeToString` boils down to [this code](
   bool MessageLite::AppendPartialToString(std::string* output) const {
     size_t old_size = output->size();
     size_t byte_size = ByteSizeLong();
     if (byte_size > INT_MAX) {
       GOOGLE_LOG(ERROR) << GetTypeName()
                  << " exceeded maximum protobuf size of 2GB: " << byte_size;
       return false;
     STLStringResizeUninitialized(output, old_size + byte_size);
     uint8* start =
         reinterpret_cast<uint8*>(io::mutable_string_data(output) + old_size);
     SerializeToArrayImpl(*this, start, byte_size);
     return true;
   This will only return false for the message being too large? Where do you see out of memory
being handled? It also looks like invalid UTF-8 does not fail serialization, but rather only
logs an error for "proto3" always and logs an error in debug mode for "proto2", see [this
example of a proto3 message](
   On the parsing side, looks like the possible cases for a false return are: `!IsInitialized()`
didn't consume entire message [[3]](,
invalid message data (e.g. invalid varint, length of a string / submessage goes out of bounds).
For invalid UTF-8, it looks [pretty complicated](
   * If the message is "proto3", it seems to always validate UTF-8 by directly calling `WireFormatLite::VerifyUtf8String`
and will then strictly fail parsing if not UTF-8.
   * For "proto2" code, it seems to call [`WireFormat::VerifyUTF8StringNamedField`](
and **only log** if it's invalid UTF-8, not fail. Also the code seems to be within a `#ifdef
GOOGLE_PROTOBUF_UTF8_VALIDATION_ENABLED` which in turns is [only enabled if in a debug build
i.e. `!NDEBUG`](
I'm a little puzzled at how you saw the logging, is your libprotobuf using debug mode? If
you're using protoc from mesos I wonder if the way it's built in mesos is hitting the same
issue as
   Here's a summary of my findings:
   **Serialization of invalid UTF-8**:
   * "proto3": serialization will succeed but invalid UTF-8 will be logged.
   * "proto2": serialization will succeed. Invalid UTF-8 will only be logged in debug mode
(NDEBUG not defined).
   **De-serialization of invalid UTF-8**:
   * "proto3": de-serialization will fail and it will be logged.
   * "proto2": de-serialization will succeed. Invalid UTF-8 will only be logged in debug mode
(NDEBUG not defined).

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

With regards,
Apache Git Services

View raw message