callback to negotiate the pixelFormat.
|fmt ||is the list of formats which are supported by the codec, its terminated by -1 as 0 is a valid format, the formats are ordered by quality the first is allways the native one |
- the choosen format
- encoding: unused
- decoding: set by user, if not set then the native format will always be choosen