Last November at the Veeam100 Summit in Prague, the Veeam Certification Team asked for volunteers from the Veeam100 group for those interested in participating as Subject Matter Experts (SMEs) to help design, develop, and write stems (questions) for the new Veeam Certified Expert+ certification exam for Veeam Data Platform v13. I was honored to be one of the SMEs selected. Fellow VMCE+ Page Moderator & Veeam Vanguard
For some of you who've been in the virtualization and BC/DR tech communities for a while, you may know I've written quite a few tech certification Study Guides for the Community over the years. Those Guides were based off VMware, Cisco, AWS, & Azure exams, and even one most recently for the Veeam VMCA v12. But, participating in the development of a vendor tech certification is an entirely different & unique challenge, one I was excited to contribute to, and one I believe worth sharing. Keep an eye out for a post from Jos who will also share his perspective on the process. So what did this process involve?
Overview
Veeam procured a 3rd party company, Alpine Testing Solutions, to drive the exam certification development process. In my opinion, this company did a fantastic job. Also, the SME volunteers selected to help with the certification exam development process → Veeam employees, MSPs, and Veeam customers really jelled well together. I don't know how Veeam does it. Whether it's deciding who is selected as a Veeam100 member by Veeam Product Strategy, or who gets selected to help develop their certification exams by the Veeam Certification team, they all seem to select individuals who work really well together.
JTA
The first phase of developing Veeam's new VMCE+ exam involved what is called a Job Task Analysis (JTA). From a basic level, this is really the in-depth process of deciding what a minimally qualified candidate (MQC) is, what expertise and foreknowledge a MQC should have before entering the exam, and what items a MQC should know about for the exam itself. It sounds basic, but is really a well thougtout and in-depth process of calloboration and discussion with the SME team. From there, the SME team talked through what all items should be required for the MQC to study going into the exam. As you can imagine, with the addition of Veeam Recovery Orchestrator (VRO) & added focus on VeeamONE (VONE) for this new exam, this process was difficult to narrow down. After about 5 days and 24 hours of virtual meeting time, we came out of this step with what everyone has come to know as the certification's Exam Blueprint.
IDW
The 2nd phase of the certification development process is what's called the Item Development Workshop (IDW). This phase requires approximately 10 days and 30 hours of time to complete. Between the completion of the JTA phase and beginning of this phase is around a 5-7 day gap. But that doesn't mean work wasn't being done. During this gap between virtual meeting sessions, the SMEs were tasked to come up with around 15-20 exam stems each from the various sub-sections of the Exam Blueprint created in the JTA phase. As one who's written quite a few Study Guides over the years, and who has managed the VMCE Study Hall page and created numerous VMCE practice questions the past 2yrs, this was really where I thought I excelled. Although I will say…it took a couple stems or so to get into a stem-writing groove. 😉 We had to have these stems written and submitted before the virtual meetings of the IDW phase began.
The stems SMEs came up with also had to meet vigorous criteria based on Veeam & industry stem writing guidelines. Some of the criteria are:
- Keep stems as short & simple as possible
- Keep stem language clear, positive, concise
- For higher level questions with bulk information, use bullets where possible instead of several sentences
- Avoid region-specific terms
- Avoid negative items (i.e. "can't not do x, y, z", etc)
- Don't make the stem "informational" or "explanatory", or have more than 1 question in the stem
- Avoid stems asking for the "best" option; be direct in what is asked
- Etc.
There are many other criterion we had to meet, but those give you at least an idea of how the SMEs had to approach writing the exam stems. During our IDW virtual sessions, we then cherry-picked all the stems, yes…one-by-one, to see if 1. the stems were accurate in what they asked in aligning with Veeam best practices/Guides/KBs and documentation, 2. met the exam writing criterion we were tasked to follow, 3. the stem met the requirement of the given Exam Blue Print section (did it ask what it was supposed to), 4. decide the relevancy of the question – yes, even after meeting the first 3 bullets, we had to decide if the question was one the MQC needed to know for the exam itself. This was really a tedious process, but hopefully for those who take the exam, you all see the fruits of the SME's labor 😊
Finish Line
After the SMEs completed the IDW phase of the certification exam development process, we handed everything off to the Veeam Certification Team. I'm not entirely clear on what they had to do to specifically finish the process. I think all the questions had to get run through again by the Certification Team to assure the stems met the criterion discussed above. Then, I believe the Certification Team assembled a group to pilot the exam to see how each stem faired. After this process…the new VMCE+ certification exam will then be released. And, I have good information that the release of the new VMCE+ exam will be coming in the next several days, so be on the lookout!
And there you have it. A high-level insight to how a Veeam certification exam is developed. I'm sure Jos would agree with me here, while the process required a significant time commitment, it was an excellent learning and incredibly rewarding experience.
Happy testing!
