National Energy Board

www.neb-one.gc.ca

Common menu bar

• Français
• Home
• Contact Us
• Help
• Search
• canada.gc.ca

Breadcrumb

Information Management Records (IMR) - Audit

Please note that certain documents in this section are available in PDF format. To inquire about receiving these documents in another format, please contact us. If you do not have PDF viewing software, you can download a free PDF viewer from the Adobe® Web site.

Information Management Records (IMR) - Audit - July 2008 [PDF 338 KB]

Table of Contents

1. Executive Summary

1.1 Introduction
1.2 Objectives
1.3 Approach

2. Interviews - Findings / Recommendations

2.1 Adequacy and Effectiveness of Training
2.2 Frequency of Use
2.3 Stability and User Friendliness
2.4 Support and Communication
2.5 Integration with QMS and Other Technologies
2.6 Ability to Maintain Restricted Information
2.7 Comfort Level with the Project Approach

3. IMR Capacity Check

Appendix I - RDIMS Implementation Interview Questionnaire

Appendix II - RDIMS Implementation - Maturity Criteria Description

Appendix III - Documents Reviewed

1. Executive Summary

The enclosed report summarizes the results of the Information Management Renewal (IMR) audit conducted at National Energy Board, in Calgary, Alberta in May/June, 2008. The audit was conducted by a contractor, Ljubica Kuraica-Siveska, under the direction of the Internal Audit and Review Services Advisor, Tony Mitchell.

The report is based on information harvested from two primary sources: participant interviews and a review of project practices, processes and related documentation. These were evaluated and analyzed to identify opportunities for improvement and develop recommendations. Two methodologies have been used in this audit:

• The first approach was to look at the current records management practices, RDIMS implementation that has been done up to date and the work that has to proceed to move the implementation to the next phase.
• The second approach, used in conjunction with the first, has been used to assess the IMR management framework capacity. The second approach is based on generally accepted best practices, and therefore provides an additional opportunity to identify gaps between current and best practices, and provides advice on how to address the gaps.

Since the beginning of the Information Management Renewal Project (January 2007), several significant goals have been achieved including:

• Project Working Group endorsement of the RDIMS application configuration and profile.
• Staffing the IM Team.
• Training and Coaching of the majority of the Board.
• Implementation and integration of RDIMS version 4.7 on user's desktops across the Board.
• System stability.
• Specialized knowledge bases designed for different Board functions.
• Commencement of IMR process engagements.
• Building relationships and process based focus.
• Compliance with Treasury Board of Canada's July 1, 2007 Policy on Information Management.

However although much has indeed been accomplished it is clear to almost everyone that much remains to be done. Considering that RDIMS is still in the early phases of adoption it is obvious that RDIMS benefits are not fully embraced. At the moment RDIMS is not being used by many employees; some of the major identified root causes are:

• File plan (structure) not developed for most of the processes. During training, users were advised to save their records in generic "To be filed" directory; however there are major concerns about the ability to locate them in future (Based on improper file naming conventions etc.).
• RDIMS not being integrated with QMS and other technologies like ESIMS, ELEXIM, Translation software etc.
• Lack of enforcement / accountability.
• No timeframes associated with user acceptance.
• Lack of communication.
• Work overload / other priorities.

As a consequence, there is a high user resistance and a perception of RDIMS being user unfriendly and time consuming. Also training retention and a loss of the RDIMS momentum are evident.

The overall conclusion is that although the project shows solid framework and sense of direction, there are still a few high risk variables that are associated with the project success. The maturity of the QMS Processes as well as the resources for QMS and RDIMS remains to be ongoing challenges. The audit identified 16 recommendations and 50 opportunities for improvement. Among those recommendations the following are in need of immediate attention:

• Close-out Implementation and Commence Operations Phase. (Revise the scope, objectives, milestones and deliverables for this phase and have them officially approved by the steering committee).
• Communicate the Operations Phase of the implementation. Communicate what the next steps are and clarify what happens with information/records storage produced by QMS processes that will not be integrated this year (Ensure that user perceptions are consistent).
• Revise the IM Policy to clarify responsibilities and accountabilities for everyone involved. (Then accountability could be monitored as part of the employee's RESULTS program).
• Formalize the QMS Engagement Process to be officially accepted and "signed-off" by the Process Owner. (Once it's signed-off RDIMS becomes the official information repository).
• Incorporate Key Performance Indicators (KPI) and Risk Factors Analysis into the project status reports to ensure accurate project control and monitoring.

Detailed findings with recommendations are discussed in Chapter 2. To enhance the overall IMR management framework additional opportunities for improvements are presented in Chapter 3.

At this time I would like to thank the NEB and a number of personnel from the various Business Units for the information feedback and assistance given to me by all involved.

Best Regards,

Ljubica Kuraica-Siveska

1.1 Introduction

The National Energy Board's Records Renewal Program was initiated in mid-2003 in response to a Consulting & Audit Canada report on records management practices at NEB. In September 2005 a second phase of the program ("The Phase 2 Project") was initiated and in the summer of 2006 a rising level of concerns about the project resulted in a decision to conduct a review and risk assessment of the program performed by Lawrence Hobbs of Chinook Solutions Inc. During the assessment it was strongly recommended that the Phase 2 Project be carefully paused while management came to grips with the significant problems that affected the very foundation of the project. In the beginning of the fiscal year 2007/2008, after the program was revalidated and proper business analysis had been developed, the subsequent implementation of Information Management Renewal program started. To prevent some of the previous scenarios happening again, it was decided that the program was to be assessed in two time frames; in May/June 2008, during the implementation of the Records Document Information Management System (RDIMS) and during the latter part of the project when it becomes fully operational.

1.2 Objectives

The objective of this report is to assess current records management practices, work of the RDIMS implementation that has been done up to date and the work that has to proceed to move the implementation to the next phase. The sets of objectives are:

• Examine the capabilities of RDIMS to collect and store permanent and transitory records.
• Examine the capability of RDIMS to collect and maintain confidential and secure records.
• Determine whether user requirements and concerns are being fully met with the software and the implementation process.
• Establish a comprehensive baseline against which the full implementation of RDIMS will be measured and compared and to provide short term and long term opportunities for improvement.
• Assess management framework capacity and compare it with generally accepted best practices.
• Ensure that the design and implementation of RDIMS is in compliance with Treasury Board of Canada's July 1, 2007 s.

The findings and recommendations are intended to provide input to the Internal Audit and Review Services Advisor, the Audit & Evaluations Committee as well as the IM Manager.

1.3 Approach

The assessment was conducted by a contractor, Ljubica Kuraica-Siveska, under the direction of the Internal Audit and Review Services Advisor, Tony Mitchell. The final report is based on information harvested from two primary sources: participant interviews and review of project practices, processes and related documentation. These were evaluated and analyzed to identify opportunities for improvement and develop recommended strategies. The work was organized into the following four stages:

Stage 1 - Documentation Review and Project Practices

In stage 1, extensive project documentation was collected and reviewed to establish the appropriateness of the project plan, the progress of the implementation to date and the potential influence of policies, practices, procedures and behaviors on the chances of success. In addition, information from the Treasury Board, other government departments and from outside sources was reviewed to determine legislative and policy requirements. Findings from previous studies and experiences and best practices of others were also reviewed. (A list of documents reviewed for this report is included in Appendix III)

Stage 2 - Interviews

30 participants were interviewed between May 21^st, 2008 and June 13^th. In order to get in-depth information, the interviewees (staff and managers) were selected from almost all of the Business Units and have various levels of involvement (some of them have already used RDIMS for 3+ years, while others are relatively new to the system). Each interview was conducted using a questionnaire and lasted between 30 min. to 1 hour. There was somewhat less participation in the interviews than originally expected; this may be explained by conflicting schedules/other commitments, resistance or simply that RDIMS is not fully implemented and not frequently used by some of them. The interview questionnaire (Appendix I) was prepared to capture the current user's perception to some of the key elements of the RDIMS Implementation.

Stage 3 - RDIMS Capacity Check Assessment

A successful IMR Implementation requires the efficient and effective transformation of business processes, technology and people skills. To assess the management framework, identify gaps between current and best practices, and provide advice on how to address the gaps, an IMR capacity check was conducted.

The objectives of the capacity check are to:

• Compare against best practices.
• Bring together all elements of the IMR management framework.
• Form a comprehensive baseline of the current maturity state, against which the NEB will gauge progress over time.

The methodology and tool used for this assignment is based on the e-Government Capacity Check, originally developed by the KPMG Consulting LP. It consists of six key elements (as presented on Table 1 below) and evaluation criteria for each key element across a maturity model scale of 1-5. (Detailed description of the capacity check and scoring model is available in Appendix II).

Stage 4 - Response Formulation (Findings and Recommendations)

In this stage, 16 recommendations and 50 opportunities for improvement were identified and preliminary mitigation strategies were recommended before the results were validated (in draft form) with the Internal Audit and Review Services Advisor and IM manager. This final report was then developed and delivered.

2. Interviews - Findings / Recommendations

The results presented in this section are based on information gathered through interviews with National Energy Board's staff and managers. A considerable amount of feedback from 30 interviewees was collected and organized into seven major areas. Closed questions from the questionnaire permitted the production of the charts presented in the following sections:

2.1 Adequacy and Effectiveness of Training

The majority of people (61%) thought that the training was "Mostly effective" and adequate. They think the training overall was good and the trainer's classroom style very interactive and user friendly. For some people the training was very informative but too basic, and it was noted that they need closer ties to how they will use it with their processes/procedures. There were comments that the online security information training was a bit dry and difficult to concentrate on.

Considering that RDIMS is not fully operational at this time (eg. File plan and process document's profiles have not been developed for most of the processes), the timing of the training for some users worked better than others.

R1: Custom developed ( process based) follow-up training sessions as part of the QMS Engagement Process will be beneficial to deal with the paradigm shift. When developing the training incorporate lots of real life examples that are relevant to the daily process activities. For example: best and worst case scenarios, tips how to prevent them…etc. (This is action in progress; IM team is already working on something similar).

2.2 Frequency of Use

Most of the people received the training and have RDIMS on their desktops; however they don't know what to do next. Some of them are even advised by their team leaders not to use RDIMS until the file plan is developed. As illustrated on the graph below, it is obvious that most of the people rarely or never use RDIMS. Some of the major causes are:

• File plan (structure) not developed for most of the processes. There are concerns about saving documents in "To be filed" directory and inability to locate them in future (Based on improper file naming conventions etc.).
• RDIMS not being integrated with QMS and other technologies like ESIMS, ELEXIM, Translation software etc.
• Lack of enforcement / accountability.
• No timeframes associated with user acceptance.
• Work overload / other priorities.
• Perceived as user unfriendly and time consuming, etc.

R2 - Close-out Implementation and Commence Operations Phase. To start with a clean plate, start the integration with the QMS processes as a new project/phase. Revise the scope, objectives, milestones and deliverables for this phase and have them officially approved by the steering committee. To reflect that, revise all pertinent documents such as:

• Project Plan
• Program Strategy
• Change Management Plan
• QMS Process Engagements, etc.

R3 - Communicate the Objectives of the Operations Phase of the implementation. Communicate what the next steps are and clarify what happens with information/records storage produced by QMS processes that will not be integrated this year. Ensure that user perceptions are consistent. ("Let's Hear It from Leadership!" section on the iWeb could be one of the options).

R4 - Revise the IM Policy to clarify responsibilities and accountabilities for everyone involved (Then accountability could be monitored as part of the employee's RESULTS program).

R5 - Formalize the QMS Engagement Process to be officially accepted and "signed-off" by the Process Owner. At the moment, as part of the QMS Engagement Process, the process owner and stewards work together with the RDIMS team to develop an acceptable file plan, profiles and file naming conventions; however there is no associated time frame when RDIMS becomes the official document's repository. Once they produce output that is acceptable and agreed by the process owner, RDIMS should become mandatory and no excuses for not using it should be accepted. (After the process is accepted, documents migration should commence and access to the network directories should be blocked).

2.3 Stability and User Friendliness

Most of the people find RDIMS very stable and didn't experience any major problems. There have been few minor glitches that are relatively expectable for this stage; however most of them are solved at this time. Unlike system stability, there is a spread between perceptions on the user friendliness (as illustrated on the graph below). Many users are overwhelmed by the profile screen and feel that it's time consuming (especially for users who periodically deal with up to 600 documents a day). On the other side, some of the users who are using RDIMS for some time commented that it takes some time to get used to it, but after that it is straight forward to use.

R6 - Simplify the Profile Screen. It took some time for the profile screen do be developed to best serve most of the users on a corporate level and the profile screen was designed based on users feedback and needs. However in almost 100% of the cases most of the sections are not used and only the mandatory fields are populated. Although there is benefit for those fields to be part of the profile, if they are not used they are just adding to the overwhelming (time consuming) perception of the profile screen. As part of the Quality Assurance function, monitor if and which parts of the profile screen are used; if they continue to be unused re-assess the possibility of removing them.

2.4 Support and Communication

Although RDIMS support is not used that much at this stage (mostly due to the fact that not many employees are using RDIMS), half of the interviewees agreed that support is available, and RDIMS team is very responsive and professional. The other half believes that there is a space for improvement, mostly referring to the communication part of it. Some of the interviewees are aware that the IM page is available on the iWeb, but they think its "buried" in the background and hard to find, while most of them didn't even know that the page exists. It was determined that many of the interviewees weren't clear who to call for a support (i.e. service desk, RDIMS team, Documentation Services etc.). Further to that, there were comments that response to some questions was never received because most likely they were sent to the wrong person/group.

R7 - Communicate that Service Desk is central support contact. All inquires/questions should go through the Service Desk (This is already specified in related RDIMS documentation; however it needs to be better communicated). Then the Service desk can forward the issue to the appropriate personnel for further assistance. To ensure there aren't any outstanding issues, this should be monitored by producing periodically reports and statistics (e.g. monthly or quarterly if more appropriate).

R8 - Make IM page "visible". RDIMS affects almost all of the NEB employees. For a project of this nature, its page should be easily accessible and be available from the main page on the iWeb (something similar like the Process DashBoard). There are great knowledge management base and support tools like tips and quick guides available, unfortunately almost nobody knows about them.

R9 - Incorporate "Communications" into the project plan and engage them to play a major role in keeping employees informed with RDIMS latest developments. (The reason it should be officially incorporated in the project plan is to allocate Communications resources and to define/monitor pertinent deliverables).

R10 - Post all IMR related Processes/Procedures on the DashBoard. Some of the benefits could be:

• Consistency with the other Business Processes.
• Increase awareness and improve communication.
• Opportunity to collect user's feedback directly.

2.5 Integration with QMS and Other Technologies

There are mixed feelings about the complexity of the RDIMS integration with the QMS Processes and other technologies; however interviewees all agree that the only way to ensure effective business integration and transformation is through a process focused approach. While some groups are confident that it will be a relatively smooth integration, other groups are finding it more complex and challenging. Most of them agree that there are workable solutions, however it will take a long time. Some of the main challenges are:

• Immature QMS processes
• Resources / Work overload.
• Integration with other technologies (ESIM, ELEXIM etc.).
• Process interactions and interdependencies.

R11 - Keep the QMS community closely involved. "Information / Records" was one of the major gaps of the QMS Audit, so the RDIMS integration with QMS will definitely leverage that score. Considering the correlation and the impact on every QMS process, the QMS Manager should be kept informed. Additionally, some of the QMS documents (eg. "Procedure Template" which has "Information/Records" section as part of the template) will be affected, so it is important that these issues be addressed, standardized and communicated to process owners / stewards right at the beginning of the integration. (Based on a recommendation provided early in the audit phase, this recommendation is already initiated).

R12 - Group the interdependent processes together in the IMR QMS Process Engagement Plan in order to prevent user frustrations of navigating between multiple environments. (In progress, already addressed in the IMR Business Case - June 17, 2008).

R13 - Communicate which QMS processes are scheduled to be integrated this year. That will allow process owners to properly plan their involvement/resources (A few of the process owners commented that they don't have any related resources planned for this year).

R14 - Develop a plan for documents sharing between RDIMS integrated and non-integrated groups / processes. To reduce confusion and frustration clearly communicate how various groups will share information /records until RDIMS become fully operational.

2.6 Ability to Maintain Restricted Information

There is absolutely no doubt that RDIMS technology is capable and reliable of maintaining restricted and personal information; however there is a significant risk of human error based incidents. At this point, the application is not fully used, so only a couple of the interviewees experienced or knows of any incidents; however more incidents are expected to happen proportionally with the increased number of employees who use RDIMS. (One of the incidents was an actual document attached instead of a reference (link) to a document, enabling some unauthorized people to view it. Luckily it wasn't of a great sensitive nature, so no damage was caused….This is one of the few possible scenarios where sensitive information could be improperly exposed, again, based purely on human error.)

R15 - All security incidents should be documented and reported to the Security Manager. It is extremely important these incidents to be further analyzed. Based on the root cause a protection measure that will prevent similar incident happening again should be developed. (Measures should be commensurate with the seriousness and number of the incidents).

R16 - Additional training that will emphasize security. Real life examples of what could go wrong should be provided to all groups that are dealing with sensitive information more frequently (eg. Human Resources, Legal Services etc.).

2.7 Comfort Level with the Project Approach

As illustrated on the graph, most of the people are "somewhat" comfortable or "neutral" with the project approach. Some of the interviewees who already initiated the QMS Engagement Process are confident that it will be a smooth integration, and they don't see any major problems; however time and resources remain to be major obstacles. Small percentage of the interviewees felt uncomfortable with the project approach, mostly due to the complexity and the size of the IMR project, work overload and the available resources. Few users felt a bit demoralized as they commenced a file plan process some time ago, but they haven't heard anything since then. As stated before, lack of communication is evident and a few recommendations are already made in the previous sections.

3. IMR Capacity Check

A successful IMR Implementation requires an efficient and effective transformation of business processes, technology and people skills. To assess the management framework, identify gaps between current and best practices, and provide advice on how to address the gaps, an IMR capacity check was conducted.

The mechanics of the Capacity Check are:

• Current capabilities are assessed based on key elements of the IMR Capacity Check, and criteria provided for each key element (Detailed Criteria Rating Descriptions are available in Appendix II).
• The capability descriptions are based on generally recognized best practices.
• A rating system of "1" to "5" is used. A high rating does not necessarily mean "goodness", but rather, formality or maturity of capability. (Level "3" is considered the norm, while the ideal rating for any area is dependent on the needs and goals of the NEB, as well as the available resources at this time).

During the assessment all findings were consolidated and current capability levels for each criterion was identified as illustrated on the graph on the next page.

A few strong elements (capabilities) such as Project Management, Relationship Management and Technology are evident, while potential initiatives that IMR could pursue to improve its management foundation and practices are strongly suggested in the following areas:

• Strategies, Plans and Policies
• Risk Management
• Benefits Monitoring
• Predictability
• Maturity Reporting

Detailed findings with opportunities for improvement are described below, in Table 2:

Table 2 - RDIMS Capacity Check Findings / Opportunities for Improvement

3.1 Strategy

Key Questions: Does the organization have a clear vision of RDIMS, and is this vision supported through the organization's strategies, plans and adequate resources? Is there senior management awareness of, and support for, the change involved in implementing the vision? Has an investment strategy been developed to fund future RDIMS projects and initiatives?

3.2 Architecture

Key Questions: Are service delivery processes sufficiently well defined and understood to support the transformation to RDIMS? Is the IM/IT infrastructure in place to manage the data, applications, hardware, software and networks for RDIMS?

3.3 Risk and Program Management

Key Questions: Has the organization adequately addressed the key risks in transitioning to RDIMS at the organization, project, and service program levels? Does the organization manage RDIMS investments proactively and monitor risks on an ongoing basis?

3.4 Organizational Capabilities

Key Questions: Do the people, tools and knowledge necessary to transform traditional Information Management service delivery exist within the organization? Are the mechanisms in place to ensure staff acquires the skills and competencies necessary to deliver RDIMS?

3.5 Value Chain Integration

Key Questions: Are customers, partners and suppliers aware of, and willing to support, RDIMS? To what extent can the organization build on existing partnerships and customer relationships to achieve greater integration of services through RDIMS?

3.6 Performance Management

Key Questions: Is the organization monitoring service metrics and customer satisfaction and adjusting accordingly? Are there mechanisms in place to measure system reliability and availability?

• Use the maturity criteria for self-assessment to gauge progress over time. (Ongoing measurement of the RDIMS maturity will allow NEB to adjust RDIMS to ensure success).
• Trend analysis should demonstrate an increasing capability to deliver customer satisfaction and significant proportion of records and documents managed through RDIMS with a high level of integration with other applications.

Appendix I - RDIMS Implementation Interview Questionnaire

Name (Optional): Business Unit:	Team: Date:
As part of the RDIMS Renewal Program Assessment, the following template will be used for a series of interviews planned between May 21st, 2008 and June 13th, 2008. The purpose of these interviews is to: Examine the capabilities of RDIMS to collect and store permanent and transitory records. Examine the capability of RDIMS to collect and maintain confidential and secure records. Determine whether user requirements and concerns are being fully met with the software and the implementation process. Establish comprehensive baseline against which the full implementation of RDIMS will be measured and compared. The results will be compiled and presented to the Internal Audit and Review Services Advisor at NEB, along with the complete assessment of the RDIMS Renewal Program.
1. Have you received training regarding your responsibilities relative to RDIMS and how to meet them? Was the training adequate and effective?
Yes Mostly A Little Not at all	Comments:
2. Are you using RDIMS for saving information?	Yes No
If Yes, how often? Daily Weekly Monthly Never Do you find RDIMS to be stable? Yes Partly A Little Not at all Do you find RDIMS to be user friendly? Yes Partly A Little Not at all	If No, please explain why:
3. RDIMS support is available and appropriate communication style is used?
Yes Mostly Needs Improvement Not at all	Comments:
4. How do you oversee the RDIMS integration with your QMS process and other technologies?
Easy Minor difficulties Major difficulties Will not integrate	Comments:
5. Do you feel RDIMS is successful in keeping sensitive and personal information restricted to the audience the information is intended?
Yes Mostly Needs Improvement Not at all	Comments:
6. How comfortable are you with the approach and plan for completing the RDIMS project?
Very Somewhat Neutral Uncomfortable	Comments:
7. What do you consider to be the major issues and challenges in the existing RDIMS implementation phase?
Comments:

Appendix II - RDIMS Implementation - Maturity Criteria Description

Appendix III - Documents Reviewed

Primary navigation (left column)

Date Modified:

2011-10-28