Some recent update in Media Foundation platform introduced a new bug related to fragmented MP4 files and H.264 video. The bug shows up consistently with file versions:

• mfplat.dll – 10.0.14393.351 (rs1_release_inmarket.161014-1755)    15-Oct-16 05:48
• mfmp4srcsnk.dll – 10.0.14393.351 (rs1_release_inmarket.161014-1755)    15-Oct-16 05:45

The nature of the problem is that MPEG-4 File Source is incorrectly time stamping the data: frame time stamps are incorrect, they seems to be getting wrong durations and increments, then quickly jumps into future… and on playback this leads to unobvious playback freezes. As Media Foundation is used by Windows Media Player, Windows 10 Movies & TV Player, the bug is present there as well.

The original report is on MSDN Forums.

Presumably it is possible to roll certain Windows Update package back, or alternatively one has to wait for Microsoft to fix the problem and deliver a new update deploying the fix.

I wrote a few posts on hardware H.264 encoding (e.g. this and the latest one Applying Hardsubs to H.264 Video with GPU). A blog reader asked a question regarding availability of the mentioned Intel Quick Sync Video support with low end Intel x5-Z8300 CPU.

[…] Intel has advertised that the Cherry Trail CPUs support H264 encoding and / or QSV, but nowhere have I seen a demo of this being used […].

What did you use to encode the video? Is the QSV codec available in the x5-z8300 for possible 720p realtime encoding? I’d like to see this checked in regards to using software like FFmpeg with qsv_h264 -codec and OBS. […]

A picture below explains how applications are consuming Intel’s hardware video compression offering in Windows.

Intel QSV includes hardware implementation of the encoder and corresponding drivers which provide a frontend API to software. This includes a component which integrates the codec with Microsoft’s Media Foundation API. Applications are to choose between interfacing the codec using Windows API – this is the way stock Microsoft applications work, and this is the way I used for video encoding development mentioned on the blog. Other applications prefer to interface through Intel Media SDK, which is an alternate route ending up at the same hardware-backed services.

Intel x5-Z8300 system in question has H.264 video encoding support integrated into Windows API and the services can be consumed without additional Intel runtime and/or development kit. The codec, according to the benchmarks made earlier, is fast enough to handle real-time 720p video encoding nevertheless the device is a budget thing.

User’s question about Alax.Info IP Video Source DirectShow filter for JPEG/M-JPEG video:

As it stands now, in order to configure the video capture source filter, I have to instantiate it […], set the properties (camera URL, and video dimensions), then save the filter state to a data array which is then compiled into the application. That doesn’t provide a feasible system for user configuration at run time: Telling our customers we need to have the IP address(es) of their camera(s) so we can build a custom version of our software for them isn’t going to be very marketable.

[…]

I looked on the Alax.Info Web site, and didn’t see anything that looks like an API for configuration. I also didn’t see anything inside […] that looks like configuration parameters. I find it hard to believe this isn’t a common requirement, so I suspect I’m overlooking something.

Is there an “easy” way to configure the parameters of the Alax.info video capture filter at run time? Where is it documented?

Is there a structure that describes the state info that is captured [into raw configuration data array]?

Are there any other considerations I’ve overlooked?

C++ and C# sample code (Trac, SVN) appears to be not easy no discover. The code snippets include PlayJpegLocation (C++) and PlayJpegLocationSharp (C#) projects/snippets demonstrating import of IP Video Source COM interfaces via COM and use them from code.

IP Video Source comes with a COM type library which is well suited for import by development environments, specifically by Visual Studio as a reference for .NET projects. To avoid bitness issues, it is recommended to have both 32 and 64 bit versions of IP Video Source installed.

C# integration is straightforward:

First, one needs to add a reference to AlaxInfoIpVideoSource type library, the same way DirectShow.NET is added to the project. The type library is imported and is available via .NET COM Interop at this point (specifically, visible in Object Browser). In code, the definitions are in AlaxInfoIpVideoSource namespace, JpegVideoSourceFilter represents a filter class, with IJpegVideoSourceFilter interface offering filter configuration.

The filter is also compatible with generic COM persistence: it implements standard IPersistStream, IPersistStreamInit, IPersistPropertyBag and other interfaces, its state can be saved to stream or property bad and loaded back.

It should be good news for those interested in hardware assisted video encoding as AMD extends offering in their new hardware and offers H.265 encoder in already well-known form factor: as a Microsoft Media Foundation Transform “AMDh265Encoder”:

# System

* Version: 10.0.14393, Windows 10, VER_SUITE_SINGLEUSERTS, VER_NT_WORKSTATION
* Product: PRODUCT_PROFESSIONAL

[…]

# Display Devices

* AMD Radeon (TM) RX 480
* Instance: PCI\VEN_1002&DEV_67DF&SUBSYS_0B371002&REV_C7\4&2D78AB8F&0&0008
* DEVPKEY_Device_Manufacturer: Advanced Micro Devices, Inc.
* DEVPKEY_Device_DriverVersion: 21.19.137.514

[…]

# Category `MFT_CATEGORY_VIDEO_ENCODER`

[…]

## AMDh265Encoder

15 Attributes:

* MFT_TRANSFORM_CLSID_Attribute: {5FD65104-A924-4835-AB71-09A223E3E37B} (Type VT_CLSID)
* MF_TRANSFORM_FLAGS_Attribute: MFT_ENUM_FLAG_HARDWARE
* MFT_ENUM_HARDWARE_VENDOR_ID_Attribute: VEN_1002 (Type VT_LPWSTR)
* MFT_ENUM_HARDWARE_URL_Attribute: AMDh265Encoder (Type VT_LPWSTR)
* MFT_INPUT_TYPES_Attributes: MFVideoFormat_NV12, MFVideoFormat_ARGB32
* MFT_OUTPUT_TYPES_Attributes: MFVideoFormat_HEVC
* MFT_CODEC_MERIT_Attribute: 8 (Type VT_UI4)
* MFT_SUPPORT_DYNAMIC_FORMAT_CHANGE: 1 (Type VT_UI4)
* MF_TRANSFORM_ASYNC: 1 (Type VT_UI4)
* MF_SA_D3D11_AWARE: 1 (Type VT_UI4)
* MF_SA_D3D_AWARE: 1 (Type VT_UI4)
* MF_TRANSFORM_ASYNC_UNLOCK: 0 (Type VT_UI4)
* MFT_GFX_DRIVER_VERSION_ID_Attribute: 1.2.3.4

This follows Intel’s H.265/HEVC hardware compression offering also available in MFT form factor:

## Intel® Hardware H265 Encoder MFT

12 Attributes:

* MFT_TRANSFORM_CLSID_Attribute: {BC10864D-2B34-408F-912A-102B1B867B6C} (Type VT_CLSID)
* MF_TRANSFORM_FLAGS_Attribute: MFT_ENUM_FLAG_HARDWARE
* MFT_ENUM_HARDWARE_VENDOR_ID_Attribute: VEN_8086 (Type VT_LPWSTR)
* MFT_ENUM_HARDWARE_URL_Attribute: AA243E5D-2F73-48c7-97F7-F6FA17651651 (Type VT_LPWSTR)
* MFT_INPUT_TYPES_Attributes: {3231564E-3961-42AE-BA67-FF47CCC13EED}, MFVideoFormat_NV12, MFVideoFormat_ARGB32
* MFT_OUTPUT_TYPES_Attributes: MFVideoFormat_HEVC
* MFT_CODEC_MERIT_Attribute: 7 (Type VT_UI4)
* MFT_SUPPORT_DYNAMIC_FORMAT_CHANGE: 1 (Type VT_UI4)
* MF_TRANSFORM_ASYNC: 1 (Type VT_UI4)
* MFT_GFX_DRIVER_VERSION_ID_Attribute: 0.0.0.3

It might look weird why would someone try Direct3D mode with a DSP, which is not supposed to be Direct3D aware, but still. I am omitting the part why I even got to such scenario. The documentation says a few things about MFT_MESSAGE_SET_D3D_MANAGER:

• This message applies only to video transforms. The client should not send this message unless the MFT returns TRUE for the MF_SA_D3D_AWARE attribute (MF_SA_D3D11_AWARE for Direct3D 11).
• Do not send this message to an MFT with multiple outputs.
• An MFT should support this message only if the MFT uses DirectX Video Acceleration for video processing or decoding.
• If an MFT supports this message, it should also implement the IMFTransform::GetAttributes method and return the value TRUE…
• If an MFT does not support this message, it should return E_NOTIMPL from ProcessMessage. This is an exception to the general rule that an MFT can return S_OK from any message it ignores.

Frame Rate Converter DSP is a hybrid DMO/MFT, which in turn basically means that its “legacy” DMO upgraded to MFT using specialized wrapper. It is not supposed to be Direct3D aware, not documented as such.

However it could presumably normalize frame rate of Direct3D aware samples by dropping/duplicating samples respectively. It could easily be Direct3D aware since it does not need, in its simplest implementation, to change the data. It is easy to see that the MFT satisfies the other conditions: it is single output video transform.

The MFT correctly and expectedly does not advertise itself as Direct3D aware. It does not have transform attributes.

However, it fails to comply with documented behavior on returning E_NOTIMPL in MFT_MESSAGE_SET_D3D_MANAGER message. The message is defined to be an exception, however DSP implementation seems to be ignoring that. The wrapper could possibly be created even before the exception was introduced in first place.

The DSP does not make an exception, returns success code as if it does handle the message, and does not act as documented.

Direct3D 11 DXVA decoding documentation lacks accuracy. The API, sadly, lacks other things too but it is a different story.

D3D11_VIDEO_DECODER_BUFFER_DESC as defined in Windows SDK:

MSDN documentation lost DataSize field, which is – ironically – the most important one along with buffer type.

Related D3D11_1DDI_VIDEO_DECODERR_BUFFER_DESC structure has both fields but the Members section has an obvious copy/paste typo:

The structure and the API itself is presumably not so popular.

Recent submissions of DirectShow and Media Foundation capture capabilities brought references to new interesting hardware.

1. Magewell USB Capture HDMI Gen 2 AKA XI100DUSB-HDMI — USB\VID_2935&PID_0001

Reports ability to capture up to 1920×1080 video at 60 frames per second in YUY2 and RGB pixel formats.

• XI100DUSB-HDMI Video Media Types (Windows 10; 10.0.15063)
  
• XI100DUSB-HDMI Video Media Types (Windows 7; SP 1)

2. Inogeni 4K HDMI to USB 3.0 Converter AKA 2A34-INOGENI 4K2USB3 — USB\VID_2997&PID_0004

Goes beyond that and promises resolutions up to 4096×2160@24, 2560×1440@60

• 2A34-INOGENI 4K2USB3 Media Types (Windows 7; SP 1)

Also, not only with Windows 10, but it seems that it can work well with older OSes like Windows 7 as well.

3. Magewell Pro Capture HDMI 4K — PCI\VEN_1CD7&DEV_0012

A PCI version of HDMI capture board with a rich offering of media types and 2K resolutions

• Video (08 Pro Capture HDMI 4K) Media Types (Windows 7; SP 1)

4. Some simpler noname HDMICap USB dongle for up to 1920×1080 YUV and RGB capture — USB\VID_EB1A&PID_7595

• HDMICap Media Types (Windows 10; 10.0.14393)

(I did not have the actual hardware, I just see reported specifications)

H.265 / HEVC Video Decoder Media Foundation has been around for a while, but using Media Foundation overall one step off straightforward basic paths is like walking a minefield.

A twenty-liner below hits memory access violation inside IMFTransform::GetInputStreamInfo “Exception thrown at 0x6D1E71E7 (hevcdecoder.dll) in MfHevcDecoder01.exe: 0xC0000005: Access violation reading location 0x00000000.”:

#include "stdafx.h"
#include <mfapi.h>
#include <mftransform.h>
#include <wmcodecdsp.h>
#include <atlbase.h>
#include <atlcom.h>

#pragma comment(lib, "mfplat.lib")
#pragma comment(lib, "mfuuid.lib")
#pragma comment(lib, "wmcodecdspuuid.lib")

int main()
{
    ATLVERIFY(SUCCEEDED(CoInitialize(NULL)));
    ATLVERIFY(SUCCEEDED(MFStartup(MF_VERSION)));
    CComPtr<IMFTransform> pTransform;
#if 1
    static const MFT_REGISTER_TYPE_INFO InputTypeInformation = { MFMediaType_Video, MFVideoFormat_HEVC };
    IMFActivate** ppActivates;
    UINT32 nActivateCount = 0;
    ATLVERIFY(SUCCEEDED(MFTEnumEx(MFT_CATEGORY_VIDEO_DECODER, 0, &InputTypeInformation, NULL, &ppActivates, &nActivateCount)));
    ATLASSERT(nActivateCount > 0);
    ATLVERIFY(SUCCEEDED(ppActivates[0]->ActivateObject(__uuidof(IMFTransform), (VOID**) &pTransform)));
#else
    ATLVERIFY(SUCCEEDED(pTransform.CoCreateInstance(CLSID_CMSH265EncoderMFT)));
#endif
    MFT_INPUT_STREAM_INFO InputInformation;
    ATLVERIFY(SUCCEEDED(pTransform->GetInputStreamInfo(0, &InputInformation)));
    return 0;
}

Interestingly, alternative path around IMFActivate (see #if above) seems to be working fine.

A while ago I posted a tool that updates a [freshly built] binary version information resource and increments build numbers for version identification and tracking.

This time it is a variant of the tool: the utility would check git details for a given repository clone and embed it as a string item of the version information resource. Regardless of version numbers, the “git log -1” text printed into binary resources makes it possible to map the binary to specific source code snapshot in retroactive troubleshooting scenarios.

The embedded information is a text put into named string item in standard VERSIONINFO resource, readable for example by Visual Studio IDE:

More to that, the application itself can read its own resources and use the information to present interactively via GUI, or include the references, such as especially SHA1 hash of respective commit, into produced log files etc.

The application takes two mandatory arguments: path to repository and the binary to patch. The rest of the arguments are optional and customize the process.

Z:\Toolbox>UpdateVersionInfoGit-x64.exe
Syntax: UpdateVersionInfoGit-x64.exe argument [argument...]

Arguments:
 help - displays syntax
 path <path> - path to cloned Git repository directory (mandatory)
 binary <path> - path to binary to be patched with file version update (mandatory)
 git <path> - path to git executable
 pretty <git-log-pretty> - git log pretty parameter string, see https://git-scm.com/docs/pretty-formats (default: format:"%H%n%D%n%ci")
 language <resource-language> - resource language for version information resource (default: 0x0409 LANG_ENGLISH/SUBLANG_ENGLISH_US)
 value <resource-string-name> - version information string name to put git log value with (default: GitLog)
 dump - print version information data block dump before and after update

The utility can be easily integrated into build process as a post-build event:

"$(SolutionDir)_Bin\Third Party\UpdateVersionInfoGit-$(PlatformName).exe" path "$(SolutionDir)\." binary "$(TargetPath)"

Download links

• Binaries:
  • 32-bit: UpdateVersionInfoGit-Win32.exe
  • 64-bit: UpdateVersionInfoGit-x64.exe
  • License: This software is free to use

Microsoft released new Windows 10 update and again there is a new tsunami approaching.

Media Foundation H.265 decoder: there is no longer “preliminary documentation” notice in MSDN article. The decoder is a substandard quality software item but it covers more or less what it has to. In particular, it indeed backs H.265 video playback including with the use of DXVA2 where applicable. It could have been an impression that technology matures and who knows maybe it will be even updated to at least the quality and feature set of H.264 decoder.

To make life more interesting and challenging Fall Creators Update does a breaking change. The changes are not yet documented, and they are important: H.265 decoder is taken away. Luckily, not completely: the decoder (along with encoder) is moved to a separate Windows Store download: HEVC Video Extension.

Even though it looks like being the same decoder, it is packaged differently and consumers might require to update code to continue decoder use/consumption.

Even though mentioned above makes at least some sense, there is also an obvious bug on top of all this: 32-bit version of the decoder is BROKEN.

The released variant of the decoder is sealed by digital signature and enforces integrity checks. This looks good with 64-bit version of the software, and in particular stock Movies and TV Player application can indeed play H.265 video in 64-bit Windows because 64-bit version of the application is used, and in turn this pulls 64-bit version of the decoder. However, 32-bit version of the DLL is broken and does not work, hence 32-bit applications relying on H.265 video decoding capabilities are going to stop working with Fall Creators Update.

The problem is apparently the integrity check because if you manage to remove that, 32-bit version of H.265/HEVC decoder is operational.

It will take some time for Microsoft to identify and fix the problem. It will be fixed though, so if it is important to have 32-bit apps functional in the part of H.265 video decoding/playback, one should rather postpone Fall Creators Update.

Further reading:

• H.265 / HEVC Video Decoder
• Microsoft H265 Video Decoder MFT missing after Windows Fall Creators Update
• hevcdecoder_store.dll cannot be loaded because it is signed incorrectly (0xc0000428)
• Windows 1709 Video Playback
• Windows 10: Fall Creators Update schmeißt HEVC-Codec raus, Microsoft zwingt Nutzer in Store (in German)

Let us break the silence with a fresh Media Foundation bug.

D3D11 WARNING: ID3D11DeviceContext::PSSetShaderResources: Resource being set to PS shader resource slot 0 is inaccessible because of a previous call to ReleaseSync or GetDC. [ STATE_SETTING WARNING #7: DEVICE_PSSETSHADERRESOURCES_HAZARD]

A Direct3D 11 enabled instance of Video Processor MFT is doing something wrong with the data and produces blacked output…

As the quoted message suggests, the problem is closely related to D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX type of textures, and – again presumably – it is a bug in the Microsoft’s MFT implementation since the problem is triggered for no apparent reason and just in some scenarios.

Apparently the internal Dircet3D 11 layer itself is capable to do the scaling because it seems to be even possible to fool the MFT with incorrect media types and get the scaling done!

The problem might be tricky to catch if the conversion is taking place inside higher level Media Foundation APIs like Media Session: it might be hard to try things with the MFT while its managed by the API and the API enforces consistency of the setup. An apparent solution to the problem (one of) is to add another custom MFT in between and copy mutex-enabled texture into a simple one. Video Processor MFT does the scaling right and efficiently when it is not confused by extravagant input.

If I recall correctly, Intel was the first vendor to supply H.264 hardware video encoder as a Media Foundation Transform (since Windows 7) and overall Intel Quick Sync Video (QSV) was the earliest widely available implementation of hardware assisted encoding. However one of the aspects of such encoding seems to be missing: update of encoder parameters during encoding session.

The task itself is pretty typical and outside Media Foundation is handled for example by libx264:

/* x264_encoder_reconfig:
 * various parameters from x264_param_t are copied.
 * this takes effect immediately, on whichever frame is encoded next;
 * due to delay, this may not be the next frame passed to encoder_encode.
 * if the change should apply to some particular frame, use x264_picture_t->param instead.
 * returns 0 on success, negative on parameter validation error.
 * not all parameters can be changed; see the actual function for a detailed breakdown.
 *
 * since not all parameters can be changed, moving from preset to preset may not always
 * fully copy all relevant parameters, but should still work usably in practice. however,
 * more so than for other presets, many of the speed shortcuts used in ultrafast cannot be
 * switched out of; using reconfig to switch between ultrafast and other presets is not
 * recommended without a more fine-grained breakdown of parameters to take this into account. */

Even though Intel’s comment on Media Foundation interface for Intel QSV is such that MFT offers a subset of the available functionality:

These hardware MFT (HMFT) are provided and distributed as part of our graphics drivers. These provide Quick sync (hardware acceleration) support on platforms. Quick Sync Video H.264 Encoder MFT is a fixed function implementation, hence there is limited flexibility to make any changes to the pipeline. Yes, HMFT distributed via graphic driver on win7 includes dx9 support, Win8/8.1 dx11 and Win10 dx12.

Update of setting as encoding goes have been a must for some time. Not only MSDN has a mention of this, e.g. here in Codec Properties detail:

In Windows 8, this property can be set at any time during encoding. Changes are applied starting at the next input frame.

But also it looks like Microsoft implements a related feature in their software video encoder without properly documenting it AND Nvidia follows that in their MFT implementation. This makes me think that certain materials exist such as reference H.264 MFT implementation, which vendors use as a sample and Intel in particular does not include this in their code.

More technical detail in my question on Intel Developer Zone site: Certified Hardware Encoder for H.264 video encoding ignores quality changes:

I am trying to identify how Intel’s hardware Media Foundation Transform for H.264 video encoding is handling quality changes during video encoding process.

Some introductory material follows that shows that what is expected to work does not really work.

Microsoft defines H.264 encoder MFT behavior here in H.264 Video Encoder and provides a software only version of the codec in their operating systems. It is expected that vendors like Intel provide their own hardware codecs as “Certified Hardware Encoders”: such encoders follow the described behavior and operating system gives them a priority as to more efficient implementation.

Intel does supply Certified Hardware Encoder for H.264 implemented in e.g mfx_mft_h264ve_64.dll (just an example, names might vary, I am referring specifically to version 7.16.10.20 in this post, the version I have as current) and this DLL along with the dependencies is Intel’s implementation of such Certified Hardware Encoder.

Certified Hardware Encoder is expected to implement certain subset of functionality defined by Microsoft:

If a certified hardware encoder is present, it will generally be used instead of the inbox system encoder for Media Foundation related scenarios.
Certified encoders are required to support a certain set of ICodecAPI properties and can optionally support another set of properties.
The certification process should guarantee that the required properties are properly supported and, if an optional property is supported, that it is also properly supported.

MSDN follows that by mentioning required properties:

The following Windows 8 and Windows 8.1 ICodecAPI properties are required:

CODECAPI_AVEncCommonRateControlMode
CODECAPI_AVEncCommonQuality
CODECAPI_AVEncCommonQualityVsSpeed
…

Even though Intel’s encoder does mostly implement the properties, there is one aspect that does not really seem to work. From the same MSDN page:

 CODECAPI_AVEncCommonQuality […]

In Windows 8, this property can be set at any time during encoding. Changes are applied starting at the next input frame.

The behavior that I see (in Windows 10 Fall Creators Update) is that Intel’s Certified Hardware Encoder is ignoring quality changes during encoding. That is, the behavior I see is that encoder keeps encoding applying the initially set properties.

More to that Intel’s implementation does not replicate Microsoft’s own codec behavior via IMFTransform.ProcessEvent call which Intel seems to be stubbing with a no-op.

[…]

It is worth saying that Microsoft’s own implementation (apart from lacking proper documentation even though the feature was announced for as long ago as for Windows 8!) does not look consistent and well thought of. Let us have a look at the simplest scenario: eAVEncCommonRateControlMode_Quality. The documentation suggests that it is:

Quality-based VBR encoding. The encoder selects the bit rate to match a specified quality level. To specify the quality level, set the AVEncCommonQuality property.

Same time, CODECAPI_AVEncVideoEncodeQP property is described as a one step lower level relative for the CODECAPI_AVEncCommonQuality property:

This property applies when the rate control mode is eAVEncCommonRateControlMode_Quality.

This property configures the same encoding setting as AVEncCommonQuality. However, AVEncVideoEncodeQP enables the application to specify the value of QP directly. If both properties are set, AVEncVideoEncodeQP overrides.

Then why the heck dynamic quality update checks AVEncCommonQuality but not AVEncVideoEncodeQP? If the latter is a priority then apparently both should have been checked, and this is not what is happening. When Intel is going to fix their MFT, they have a good chance to make it right.

Designed with speed in mind, Rainway is tuned to avoid impacting the performance of your game. Enjoy 60FPS streams with super low-latency gameplay. Rainway is launched beta-version today offering a server for self-hosting and HTML5 web client. The server component is designed to stream using state-of-the-art technology and performance. On client Rainway offers quality remote gaming experience in Chrome and FireFox web browsers.

• Rainway: Play Your Favorite Games on Any Device
• Rainway, the Next-Gen Game Streaming App for PC, Enters Public Beta
• Prepare For Launch: “As we approach January 20th we want to cover a few things so you can prepare and enjoy our beta…”

A few days earlier Rainway released a trailer which features a vision on future of gaming in the world of variety of devices and fast network. Today’s beta launch is the first Rainway’s step towards putting gaming online.

Feel free to register and enjoy the new experience. Also, even though it is about gaming you are not limited to games: the technology remotes you into a workstation in general and makes remove desktop accessible via general purpose browser.

For already years Nvidia’s H.264 video encoder Media Foundation Transform has been broken giving REGDB_E_CLASSNOTREG “Class not registered” failure in certain circumstances, like main display is not the one connected to Nvidia video adapter.

As simple as this:

CoInitialize(NULL);
MFStartup(MF_VERSION);
class __declspec(uuid("{60F44560-5A20-4857-BFEF-D29773CB8040}")) CFoo; // NVIDIA H.264 Encoder MFT
CComPtr<IMFTransform> pTransform;
const HRESULT nCoCreateInstanceResult = pTransform.CoCreateInstance(__uuidof(CFoo));
// NOTE: nCoCreateInstanceResult is 0x80040154 REGDB_E_CLASSNOTREG

The COM server itself is, of course, present and registered properly via their nvEncMFTH264.dll, it’s just broken inside. Being called straightforwardly via IClassFactory.CreateInstance it gives 0x8000FFFF E_UNEXPECTED “Catastrophic Failure”, and so it also fails in Media Foundation API scenarios.

This is absolutely not fun, Nvidia!

A few updates to DirectShowReferenceSource module, it’s Media Foundation video Media Source related part today.

First, the video media source is now handling restarts from paused state correctly and resumes frame generation from proper position (not from zero as before).

Second, the video media source is now Direct3D 11 aware. That is, when it participates in Direct3D 11 enabled topologies, the media source generates the video frames using DXGI render target variant of Direct2D (see ID2D1Factory::CreateDxgiSurfaceRenderTarget for details) and delivers them downstream as textures. This is, in particular, useful to those who needs a signal to fit to Direct3D 11 aware transforms and renderers such as DX11VideoRenderer. Specifically, being connected to DX11VideoRenderer the video media source features  GPU-only video playback.

Download links

• Binaries:
  • 32-bit: DirectShowReferenceSource-Win32.dll
  • 64-bit: DirectShowReferenceSource-x64.dll
  • Sample code: SVN, Trac
  • License: This software is free to use

Being complicated for many things, Media Foundation is still quite simple for the basics. To capture video with Media Foundation the API offers Source Reader API which uses Media Foundation primitives to build a pipeline that manages origin of the data (not necessarily a live source as in this example, but can also be a file or remote resource) and offers on-request reading of the data by the application, without consumption by Media Foundation managed primitives (in this aspect Source Reader is opposed to Media Session API).

The simplest use of Source Reader to read frames from a web camera is simple enough to fit a few tens of lines of C++ code. Sample VideoCaptureSynchronous project captures video frames in the form of IMFSample samples in less then 100 lines of code.

Friendly Name: Logitech Webcam C930e
nStreamIndex 0, nStreamFlags 0x100, nTime 1215.074, pSample 0x0000000000000000
nStreamIndex 0, nStreamFlags 0x0, nTime 0.068, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 0.196, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 0.324, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 0.436, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 0.564, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 0.676, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 0.804, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 0.916, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 1.044, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 1.156, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 1.284, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 1.396, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 1.524, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 1.636, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 1.764, pSample 0x000002CAF3805D10
nStreamIndex 0, nStreamFlags 0x0, nTime 1.956, pSample 0x000002CAF3805D10
...

VideoCaptureSynchronous project does not show what to do with the samples or how to request specific format of the samples, it just shows the ease of video capture per se. The capture takes place synchronously with blocking calls requesting and obtaining the samples.

Media Foundation API is asynchronous by its design and Source Reader API in synchronous mode hides the complexity. The blocking call issues a request for a frame internally, waits until the frame arrives and makes it available.

Source Reader API does offer asynchronous model too, making it available again in as simple as possible way.

VideoCaptureAsynchronous project is doing the same video capture but asynchronously: the controlling thread just starts capture, and frames are delivered via a callback once they are available, on a worker thread.

So when does one use synchronous and asynchronous?

Even though synchronous model results in cleaner and more reliable code with less chances for a mistake, and the gains over asynchronous model in most cases can be neglected esp. to those who are interested in beginner material like this post, video capture is real time process where one doesn’t want to block controlling thread and instead receive the frames out of nowhere as soon as they are ready. Hence, the asynchronous version. Asynchronous still can be simple: VideoCaptureAsynchronous is already more than 100 lines of code, but 120 lines might be also okay.

Download links

• Source code:
  • VideoCaptureSynchronous: SVN, Trac
  • VideoCaptureAsynchronous: SVN, Trac
• License: This software is free to use

This is basically a cross-post of StackOverflow answer:

How do I encode raw 48khz/32bits PCM to FLAC using Microsoft Media Foundation?

A few years ago the question was already asked: DirectShow Spy: Who Sent EC_ERRORABORT?. The spy already attempted to log a call stack in order to identify the sender of the message. However overtime the tracing capability stopped functioning. There were a few reasons and limitations of internal stack unwinder specifically resulted in inability to capture the information of interest.

It is still important once in a while to back trace the event sender, so now it is time to improve the logging.

Updated DirectShow Spy received an option to produce a minidump at the moment of Filter Graph Manager’s IMediaEventSink.Notify call. The advantage of minidump is that it can be expanded retroactively and depending on minidump type it is possible to capture sufficient amount of details and trace the event back to certain module and/or filter.

The image below displays an example of call stack captured by the minidump. For the purpose of demonstration I used EC_COMPLETE instead though without waiting for actual EC_ERRORABORT.

The call stack shows that event is sent by quartz.dll’s Video Mixing Renderer filter’s input pin as a part of end-of-stream notification handling, which in turn has a trace of video decoder and media file splitter higher on the call stack.

The minidump generation is available in both 32 and 64 bit versions of the spy.

To enable the minidump generation, one needs to add/set the following registry value (that is, set ErrorAbort MiniDump Mode = 2):

• Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Alax.Info\Utility\DirectShowSpy
• Key Name for 32-bit application in 64-bit Windows: HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Alax.Info\Utility\DirectShowSpy
• Value Name: ErrorAbort MiniDump Mode
• Value Type: REG_DWORD
• Value: 0 Default (equal to 1), 1 Disabled, 2 Enabled

Then, to control the type of generated minidump file the following registry value can be used:

• Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Alax.Info\Utility\DirectShowSpy\Debug
• Key Name for 32-bit application in 64-bit Windows: HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Alax.Info\Utility\DirectShowSpy\Debug
• Value Name: MiniDump Type
• Value Type: REG_DWORD
• Value: respectively to MINIDUMP_TYPE enumeration; default value of zero produces MiniDumpNormal minidump

Some useful minidump type flag combinations can be looked up here: What combination of MINIDUMP_TYPE enumeration values will give me the most ‘complete’ mini dump?, and specifically the value 0x1826 expands to the following:
MiniDumpWithFullMemory | MiniDumpWithFullMemoryInfo | MiniDumpWithHandleData | MiniDumpWithThreadInfo | MiniDumpWithUnloadedModules – this gives a “complete” output.

That is, to put it short, to enable .DMP creation on EC_ERRORABORT, the following registry script needs to be merged in addition to spy registration:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Alax.Info\Utility\DirectShowSpy]
"ErrorAbort MiniDump Mode"=dword:00000002

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Alax.Info\Utility\DirectShowSpy\Debug]
"MiniDump Type"=dword:00001826

DirectShowSpy.log file will also mention the minidump file name once it is generated.

Download links

• Binaries:
  • 32-bit: DirectShowSpy-Win32.dll (1.0.0.2134)
  • 64-bit: DirectShowSpy-x64.dll (1.0.0.2133)
• Shortcuts to Exported Functions: Helper .BAT files
• License: This software is free to use
• Installation Instructions: Original post

Media Foundation version of Reference Signal Source is UWP friendly.

It can be played using stock MediaElement control (whether it is a C# or JS project), is also Direct3D aware and friendly and as such integrates well with the MediaElement. Then on top of that the whole bundle is even WACK friendly and is ready for new adventures.

Not the first, not the last. A Direct3D 11 enabled Media Foundation transfer fails to transfer sample attributes while doing the conversion.

Why attributes are important in first place? Because we can associate data with samples/frames and have them passed through attached to specific frame as the conversion goes and as the data transits through the pipeline.

There is no strict rule whether a transform needs to copy attributes from input to output samples. Attributes are flexible and in this case it’s so flexible that it is not clear what the transforms actually do. Microsoft attempted to bring some order with MFPKEY_EXATTRIBUTE_SUPPORTED property. Let us have a look at what documentation says about the processing model:

The input samples might have attributes that must be copied to the corresponding output samples.

• If the MFT returns VARIANT_TRUE for the MFPKEY_EXATTRIBUTE_SUPPORTED property, the MFT must copy the attributes.
• If the MFPKEY_EXATTRIBUTE_SUPPORTED property is either VARIANT_FALSE or is not set, the client must copy the attributes.

Words “client must copy the attributes” should be read as this: MFT does not give a damn about the attributes and go copy them yourself the way you like.

Needless to say that Video Processor MFT itself has not faintest idea about this MFPKEY_EXATTRIBUTE_SUPPORTED attribute in first place, and so is the behavior it defines.

Microsoft designed Video Processor MFT as a Swiss army knife for basic conversions. The MFT has zero degrees of customization and has multiple code paths inside to perform this or that conversion.

All together it means that small bugs inside are endless and MFT behavior is not consistent across different conversions.

So I approached the bug itself: unlike other scenarios when the MFT does pixel format conversion it fails to copy the sample attributes. I feed a sample with attributes attached and I get output with zero attributes.

In my case the workaround is this a wrapper MFT that intercepts IMFTransform::ProcessInput and IMFTransform::ProcessOutput calls and copies the missing attributes.