Project Spotlight

DHS CBP Cargo ACE Development, Migration, and Maintenance

DHS U.S. Customs and Border Protection Office of Information Technology Cargo Systems Program Directorate’s ACE Development, Operations, & Maintenance

For Cargo Systems Program Division’s (CSPD) ACE program, we provide IT services to develop, migrate, and support cargo system applications, leveraging Agile development methodologies. CSPD is a highly complex program. The Cargo Production applications provide direct support to the CSPD mission of enforcing trade laws and securing the borders while facilitating legitimate trade. For CSPD this means processing 66,000 conveyances daily, collecting over $30B annually, and include a significant number of diverse stakeholders including the CBP Offices of Field Operations and International Trade, the public trade community, Participating Government Agencies (PGAs), and foreign governments. The CSPD application portfolio includes 105 applications. Through Trinnovate agile teams and effective use of SecDevOps techniques we have established 3 weekly deployment windows and average over 55 completed deployments monthly. This allows flexible mission responsiveness while ensuring the stability and quality needs of mission critical applications are met.

Project Management

Effective alignment to respond to evolving needs is the foundation of our ACE Project Management
efforts. Our team has established a culture of collaboration, partnership, and investment between our team and CBP leadership that is operationalized by a few critical activities:

We provide bi-weekly briefings to the CSPD Executive Director (XD) and his direct reports to provide an in-depth review of project status, upcoming delivery dates, improvement initiatives, and overall funding status.
We conduct bi-weekly meetings between OIT & Office of Trade leadership to define and prioritize our program backlog, ensuring alignment on priorities.
We conduct daily stand ups with the Deputy Executive Director (DXD) and Director of Cargo Systems Development to review any outstanding delivery risks or issues and ensure effective real-time alignment of Trinnovate, CSPD and stakeholders.
We ensure we have the right staff, with the necessary skills, at the right time. Through close alignment and planning with CSPD and program leaders, we anticipate potential staffing gaps in upcoming increments and staff in advance. Through our extensive experience at CBP we also understand what’s necessary to ensure we provide clearable staff.

Operations and Maintenance, Performance Upgrades, and Enhancements

We manage our agile teams and individual sprints to balance capacity for operations & maintenance activities as well as new development initiatives across each of our Agile teams as prioritized by our Product Owners and OIT program leadership. Key development and O&M activities include:

Monitoring and responding to Remedy tickets submitted on the CSPD applications.
Our DevOps team tunes our system alerts coming from OEM, queue depth alerts, AppDynamics, and Splunk to make them meaningful, provide a high level of coverage across ACE applications.
We have really worked to enhance automated test coverage for our applications, by establishing a dedicated team to drive test automation across our Agile teams, through development of new scripts as well as mentoring developers and testers. The result is a robust set of functional regression tests run daily, with results automatically emailed to our Agile teams.
One of the most important areas of development and maintenance are interfaces. We have established and maintain XX interfaces between ACE applications and external systems, including ATS. With over 30 years of experience modernizing CSPD ACE & PSPD TECS systems, our team has unique subject matter expertise across a broad set of CBP functional processes. This expertise is critical to effective integration and has been leveraged to establish the existing interfaces between ACE & TECS to TASPD to share this data used in downstream targeting.

Cloud Migration

We have migrated 27 applications to the CBP AWS Cloud East (CACE) environment, and an additional
16 applications have been migrated to CSPD’s Docker container environment. Over the next 3 months
we will migrate an additional 50 applications to either the CACE or Docker environments. We
accomplished this by engaging all Agile teams and a variety of other program support resources through
the following strategies to help us scale, mitigate the risk and facilitate communications:

New Developers are assigned to our migration tiger team, where their first assignment is to support the migration of one application out of WebLogic mentored by our Program Technical Lead. Once complete, they have gained needed experience working with the technologies and design patterns, they then join an Agile team to apply this knowledge to accelerate the migration.
We established a Pre-Production environment in CACE & Docker certified to store Production data and scaled to handle representative Production volumes. We perform a two-part test in this environment before go-live. First, we complete performance testing of the migrated applications to verify performance under production loads; then we perform data comparison testing where we compare that a set of messages processed by both the existing Production application, and the migrated version of the application, return the same results.

Performance Innovation

To optimize the performance of our Agile teams at CSPD, we established a robust DevSecOps team that
is focused on Production availability and monitoring, expanding the use of our CI/CD pipeline, and
supporting release management. Organizing these resources in a separate team allowed us to improve
both availability and efficiency and we’re able to better adjust and accommodate changing priorities. Let
me share a few examples of the impact this team has had:

All Teams have fully migrated all ACE applications over to automated CI/CD pipelines with code repositories in Stash and/or Jenkins. The last migration was for our oldest set of ACE applications, built nearly 15 years ago, which still used Dimensions as their code repository and didn’t employ automation for their deployments. Our team successfully migrated all of the code from Dimensions to Stash, implemented automated build and deployments for all environments, and integrated the execution of automated unit tests into the build process.

Our DevOps team supports our release management process, by creating Change Requests (CRs) for
upcoming releases and managing the implementation of these CRs during our planned outage windows.
Always looking for opportunities for our teams to gain efficiency, our DevOps team developed a custom
utility to automate the CR creation by integrating JIRA and Remedy to automatically generate all of the
artifacts necessary for the CR. Reducing CR creation time for 4-8 hours per CR to minutes.

Want to know more?

CORPORATE SUMMARY

Trinnovate is a SBA 8(a) Certified (Pending) joint venture between Impyrian, C. Mack Solutions, and Potomac IT Group based in Fulton, Maryland. We are a multifaceted technology firm dedicated to helping the U.S. Federal Government, state and public sector through Security, IT and Program Management solutions.

CAPABILITIES

Application Development

Artificial Intelligence

Big Data

Business Automation Solutions

Business Intelligence Services

Business Operations Support

Cloud & Virtualization

Cyber Security

Data Visualization

DevOps

User Interface/User Experience