Xentity has a blast at our 2nd Culture Jam supporting Intercambio

Xentity recently supported one of the Summer Events hosted by Left Hand Brewing Co. which specifically raises money to benefit Intercambio Uniting Communities: Culture Jam 2015. The event is truly a fun summer night event bringing together many cultures, various music, and family friendly. The folks at Left Hand and Intercambio put on a great show, raising money to support an […]

How Understanding Complexity may help you change the game

In Data and IT, there are various levels and classes of complexity which our design methods need to adapt Choosing our methods for consulting in design, architecture, and strategic consulting need to understand the environments. For instance, Enterprise Architecture came of age during the heyday of MIS in the 1980s as standalone applications on mainframes […]

Delivering Open Data in Bulk on the Cloud

Blog post
added by
Wiki Admin

We just finished some work for a large National Government data provider who measures their number of files in the millions, records in the tens to hundreds of millions, and storage in sub-petabyte. Below is the obfuscated general requirements if you were to be looking to deliver your bulk data in the cloud : Storage requirements, access, methods, discovery, communications, and applications.

These requirements have been generalized or completely redacted or some cases, added to, to allow for all in Government Open Data delivery with large public datasets to consider. This is simply the business requirements, and not considering the technologies, vendors, cost models, capacity planning, etc. That was done separately.

1.    Storage – Storage supporting file form factors including

Investigate the free public data set clearinghouse areas like http://aws.amazon.com/publicdatasets/ or on Azure, etc. 

Consider  various form factors of files or services

  • Gigabyte Size Files
  • Medium Size Files, but totals more than Gigabyte Size Files
  • Many Terabyte or Gigabyte files that have been broken into medium files for transfer
  • Millions of small files usually delivered in buffered stream
  • Data-driven file delivery via services
  • Terabyte Files only deliverable via Sneakernet Import/Export

2. User Access – easy access for users to copy files to target environment

Public Read-Only Users should not be required to have to pay for access to end-solution (i.e. should not require user to have cloud account on hosted solution)

Internal Users will require access to private directories for files not or yet to be publicly released files (i.e. in response to emergencies, access to licenses data, interim work products)

Internal Users will benefit from lower-latency access than public users. Solutions such as cached volumes, integration with on-premise IT and cloud environment, and secure file transfer.

3.    Multiple Access Methods – Service, Download, Media, Cloud-to-Cloud

Users will look to have data provided in bulk one of three ways: Web Service, Bulk Media, or Cloud to Cloud

Admins should have access user traffic statistics for viewing, exporting statistics logs, and calling statistics logs via hosted applications.

User pulls a directory, set of directories, set of files or a mix via online web access via HTTP, REST, FTP, UDP, or SCP.

Learn about high-performance file transfer solutions are possible such as Edge Network publishing to move closer or supporting high-performance file transfer such as UDT (UDP-based data transfer protocol)

For faster and likely larger file requests, User requests a directory, set of directories, set of files or a mix to be put onto storage device by the service provider and the device is delivered back to user.

For faster and likely larger file requests, User requests a directory, set of directories, set of files or a mix to be put onto storage device by the service provider and the device is delivered back to user. Bulk Media minimum specifications for external hard drives

Users who have existing cloud accounts for storage or who have virtual machine processing points on the cloud, will make requests or will pull a directory, set of directories, set of files or a mix Data pushed to the users cloud point

4.    Discovery – increased visibility and discovery of staged products in catalogs and search engines

Data Products are usually downloaded via keyword, geospatial or temporal product discovery applications based on filtering their search, creating an order, and downloading the products in small group.

Public file directory listing should be discoverable and optimized for discovery by search engines

Public collections should be discoverable and optimized for discovery by search engines

Explicitly demonstrate how bulk data registrations will be discoverable and registered in both Sciencebase.gov and data.gov

Catalogs should be able to pull or push harvest public FGDC, ISO-19115, or RDF metadata of files in the directories for transaction or bulk loading into their catalog.

File Directory Listing can be queried via open-standard discovery service to assist in developing a download filter list.

The National Map can be discoverable in proposed service provider catalog, but the catalog reference needs to follow the metadata provided along with each file with at minimum presenting source, created date, updated day, title, basic description, and the provided DOI link for the file or directory.

Service Provider should be able to be support being called via a Digital Object Identifier

5.    Publishing– support batch file release updates for thousands of files monthly.

Consider if publishing and updating files within datasets incrementally, and will require service or bulk media methods to update the datasets.

Files published will be stored in original formats.

Updates are expected to be updated monthly at no more than on average 10% of files or file storage.

Updates to files should be logged to trigger notifications to subscribed users.

File updates should be able to maintain success and parity check status.

Offline File transfer should support processing of delivered storage devices with clear instructions

Online upload transfer per storage unit (i.e. per gigabytes) should not have transfer charges akin to transactional charges to bulk download area

Online upload should have high performance data transfer capabilities such as UDT (UDP-based data transfer protocol) for between on-premise data and cloud.

Moving from cloud to cloud, i.e. if moving from transactional area to public dataset hosting area, should have very high-speed transmission speeds and should consider location proximity issues.

6.    Notifications – providing ways for users to subscribe to staged product files update notifications

Users can subscribe to changes to directory, sub-directory, or specific files

Users can be notified of such changes via push notifications via such ways as per change, daily changes, RSS updates, or other notification techniques.

Users can use the notifications as ways to request the bulk file updates

7.    Download API  – Supporting applications or including applications that help the user download in bulk

Have a download API that can be controlled by api.data.gov which can uniquely identify, provide HTTP access to via GET parameter in a URL query, support an hourly limit of number of requests per hour based on API Key settings. If api.data.gov rate limit is exceeded, an HTTP status code of 503 should be returned.

3rd party applications should be able to support HTTP, REST, FTP, or SCP calls.

Software Development Kit access (java, python, .NET, PHP, etc.) access should be allowable as well.

The file download should be able to support multiple file requests, allow for parallel downloads, handle restarting partial download file requests, and governor anonymous volume requests

Peer-to-Peer solution support (i.e. such as BitTorrent) must comply with Federal Regulations.

Identify what, availability, and cost for User Training and Sanctioned or third-party consultants for Software Developers is available

8.    Applications – Support the end user experience for unzip files and load into geospatial database

If the user will received multiple zipped files that will require the user to click each link to download, unzip each file, and then load each file using the provided metadata manually into a database, can this be automated

Vendor can create premium either accelerator, increased access or additional formats are part of the delivery if branded separately as a vendor branded product and as long as there is one version that is published clearly marked as Authoritative Government as published and controlled by such in its original published form.

 

Remembering Henry Wang

Blog post
edited by
Matt Tricomi

Henry Wang

Technology Architect Senior Consultant 

With great sorrow we pass on that our friend and colleague, Henry Wang, on November 8, 2014 passed losing his battle with Leukemia. Henry impacted many of us at Xentity, teaching us values, leadership, and impact.

In 1999, he was my director and mentor working on projects for Sapient Corporation from United Airlines, Star Alliance, Navigant International and more. He helped me grow my architecture skills on communication, patterns, capacity planning, issues triage, deducing client needs into concepts, and many more. More so, he taught principles of values, ethics, and leadership.  One of the great premises Henry provided was “to get to the value of what someone needs, think one level higher than you thought you needed, and you may get it.”

We traveled weekly together back and forth between Boston and Chicago, Boston and Denver, Denver and Memphis, and beyond, for months. He provided valuable knowledge, and over time, would reciprocate the same. Henry and I nearly perished on September 11, 2001, but hours and a day before respectively, the UA 175 Boston to LA flight plans were changed a day due to pushing a meeting. After conversations with him, family, and very few others, two weeks later, Xentity was founded after discussions with many folk, including Henry’s guidance, about making impact.

Later in 2002, Henry joined Xentity’s first project in travel sector as an independent consultant himself. From that, a mutual partner saw his talent, and he became the CIO at OneTravel. We kept in close touch over the years, seeing his kids grow, seeing my family grow, discussing business but even when we caught up in Texas, Boston, or recently over gchats while he was in the hospital, our discussions on transformation, life, family, faith, leadership, and more carried on like we still were traveling together.

Those times will be missed by us, and his family.

-mt

 

 

How to reduce time while increasing impact for complex project analysis

Blog post
added by
Wiki Admin

Business Leaders… You are asked to lead your organization in becoming more flexible, driving down cost, reducing cycle time, and improving services to citizens across the board. In short, you are asked to do much more with way less. These are some specific Agency Performance Objectives:

  • Process Simplification
  • Standardized Procedures
  • Consolidated and Streamlined Bureau Processes, Data, Systems, Technology
  • Value-Enhanced Processes that eliminate unnecessary burden on the Citizen
  • Collaborative solutions that encourage Partnerships
  • Integrated architecture processes that facilitate knowledge transfer and reuse between business, data, application, and technology components

Prove the value through analysis to integrate pilot efforts into your broader program evolution. Prove the efficiencies, quality, and scalability sought can work – and under what factors, budget implementation, workforce impacts, data maturity, and infrastructure changes. Making sure that you can see a clear line of sight between your stakeholders goals, product and service portfolio, your concept of operations, and impacts to your resources – partnerships, data, workforce, and IT.

Our Rapid Design and Analysis Services approach can help you meet these objectives. We can support existing operations, projects, new projects, or new strategies.

Which does your organization need to focus upon for its modernization efforts?

 Read more about our Our Change Services Concepts

Sample Rapid Design Services

  • Agile Project Initiation Planning Support
  • Business Strategy Development
  • Business Process Analysis
  • Capabilities Assessment
  • Project Risk Assessment
  • Business Transition Planning
  • Business Solution Design
  • Data and Information Design
  • Business Modernization Blueprint and Implementation
  • Custom Lab Services

These types of tactical or strategic analysis or project implementation engagements should be targeted to have recommendations, transition plan, and scoped budget accepted at executive sponsor level within 3 to 6 months.

Agile Project Initiation Planning Support:

Your project plan has received approval yet you lack qualified or trained personnel to begin the implementation tasks.

The Project Initiation Support Service will assist you in forming your Core Implementation Team as well as in developing the project plan, project charter, communication strategy, funding strategy, and risk management plan.

Let us help you help you get your change effort started or back on track with our two-day workshop. This effort will be a rapid implementation planning effort that will seek to get momentum for moving forward. The workshop approach focuses on gaining consensus across your team, get the common vision, high-level, concept of operations, and priorities and dependencies for key requirements and milestones.

The effort usually requires a few weeks of preparation with the sponsor and reachback to your team, a two-day very rapid and intense workshop effort with your team, and ends with an action plan and all workshop deliverables polished after real-time capturing. The goal of the workshop, whether strategic, tactical, or technical in scope, will be to build a roadmap for the future to establish a clear “line of sight” from executive to users through costs, development, and support for your change effort. MBT:7, 10b

Business Strategy Development

Your organization needs to reevaluate its strategic direction due to changes in technology, policy, or regulations. Globalization requires that you better connect with your customers. Federal mandates require that you more effectively measure how well your organization is performing.

The Business Strategy Development Service will analyze your existing customer community and determine the exact needs and wants of that community. This service helps to bring clarity to goals and objectives, lays out an approach to reach those goals and objectives, and identifies corresponding performance objectives. Your strategy includes steps to be taken to reach your desired outcomes. MBT:1, 2, 7, 12

Business Process Analysis

Your organization has a clear mission and understands the needs of its customers, but you feel that it isn’t able to operate as efficiently as possible. Your staff is stretched too thin or is slipping in the delivery of services to your customers. Another symptom could be that benchmarking indicates that your process may be too costly.

The Business Process Analysis Service will assess how well your organization is performing by evaluating the products and services you are providing to your customers. It identifies the gaps between your organization’s objectives and your products and services, business mandates, and the needs of your customers. Once products and services are well defined, this service analyzes your business processes to determine cost factors and evaluate overall efficiency and value. Ultimately, the service provides a proposed core process that includes industry best practices as well as enhancements that result from process model simulations. It also provides guidance for implementing the needed and approved changes so that you can actually see the benefits of the analysis. MBT:1, 2, 3, 8

Capabilities Assessment

You would like an objective assessment of your organization’s existing business climate, including an assessment of current leadership, staff, organizational structure, information, and/or existing technology solutions.        

The Capabilities Assessment Service will analyze the skills and capabilities of your staff in relation to your mission requirements. The service will also analyze the leadership and current organizational structure in relation to your mission requirements as well as the specific needs of your staff. Once the existing business climate has been assessed, this service will analyze the existing information availability and exchanges as well as the technologies and applications that you are currently using to meet your mission. Overall, this service is designed to assess your operational capacity to meet customer demands. MBT:1, 2, 3, 4

Project Risk Assessment

An existing project of yours has been challenging with respect to technology, management, and/or funding. Unknown risks continue to emerge, forcing the management team to be reactive rather than proactive. This has resulted in senior management concerns that the project is at risk.

The Project Risk Assessment Service will identify and document overall risks by looking at all facets of your project. The service will document these risks and develop mitigation strategies and provide guidance to eliminate or accommodate the risks. MBT:2, 3, 4, 7

Business Transition Planning

Your organization is faced with a large-scale change either in leadership, vision, or a new technology implementation that will impact the way you do business. You know that this large-scale change will affect your people, processes, and organizational structure but you need help planning for that change.    

The Business Transition Planning Service will analyze the major changes that your business organization is facing, and how those changes will impact your strategy, processes, people, and/or technology resources. Since most major change events will actually impact many facets of your business, this service will take a robust and broad look at all areas that will most likely be impacted by the change. This service is focused on equipping you with a strategy that accomplishes a phased implementation of change components resulting in a pre-determined future state. MBT:1, 2, 3, 4, 7

Business Solution Design

Your organization has a clear challenge in performing tasks related to one of your mission areas. You feel that your organization would benefit from an automated solution (new system) to meet this mission challenge. 

The Business Solution Design Service is focused on a specific need or problem within your business organization. Specifically, it will analyze those aspects of the business that contribute to the problem. Additionally, it will analyze scenario-based alternatives that will simulate the problem with the use of technology. The service will define required technology components based on needed business features. MBT:1, 2, 3, 4, 7, 10a, 11

Data and Information Design

You feel that your organization could be more effective in sharing internal information, or in sharing data with other organizations. You know that the data you need is out there, but your organization can’t seem to find it, get it organized, or use it.    

The Data and Information Design Service will analyze your current and future data needs, determine whether the required data should be independently gathered and stored, and will assist in defining the steps to implement any required data-sharing relationships. This service is focused on producing actionable recommendations to improve the way data flows within and between DOI business organizations. MBT:1, 7, 9

Business Modernization Blueprint and Implementation

Your organization needs a make-over starting from a thorough customer analysis, thinking through the organization’s strategy and processes, understanding information needs, and then recommending the right technology solutions for the future.      

The Business Modernization Blueprint and Implementation Service is the most robust and thorough analysis service provided by the lab. This service begins with detailed customer analysis and extends into developing new goals and objectives for the business. The service then takes a closer look at the products and services for the business, followed by an analysis of needed and existing processes, skills, and technology solutions. This service provides guidance for the development of a Modernization Blueprint and its implementation. MBT:all steps

Custom Design Services

Your transformation support needs are cross-cutting, complex, and requires more combinations of discrete intermediate staffing support, and definition will come over time.