PROJECT

AI services for radiology

Goal:

Scientific research of the possibility of using decision support methods in the healthcare system of the city of Moscow based on the results of data analysis using advanced innovative technologies

Location:

Moscow region, Russia

Base:

Moscow regional RIS (ERIS)

550
MR MG CT XR

units of the outpatient and inpatient departments:

365

radiologists in the system

160 k
per month

radiology studies

44 k
per year

breast and lung cancer screening studies
Plans: extension to the national level

Use Cases

Oncology
Lung cancer. CT/LDCT
С34
Breast cancer. MG
С50
Neoplasm of adrenals. CT/LDCT
D44
Neoplasm of liver. CT/LDCT
C22
Cardiology
Coronary calcinosis. CT/LDCT
I25
Aortic aneurysm. CT/LDCT
I79
Paracardial fat. CT/LDCT
I27
Dilation of pulmonary trunk. CT/LDCT
I25
Chronic diseases
Change of liver density. CT
K70-K77
Vertebral fracture (osteoporosis). X-RAY/CT/LDCT
M80
Pulmonology
Pathology. X-RAY
J09-J18
Tuberculosis. X-RAY
A15
Neurology
Multiple sclerosis. MRI
G35
Emergencies
Cerebral hemorrhage. CT
I60-I64
Limb fractures. X-ray/CT
S10-T07
Skull fractures. CT
S10-T07
- Current
- Future

How to apply?

Preparation

Fill in and submit the questionnarie via e-mail

Check the suitability of your AI service solution

Check out quality requirements

Apply for the experiment

Integration

Conclude an agreement

Check technical requirements

Integrate your service with ERIS

Calibrate your service on the labeled dataset (historical data)

Prospective analysys for studies

This is the main stage of the Experiment

Your AI service analyses studies in real time

The Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies provides for:

  • Diagnostic accuracy evaluation
  • Study audit by expert doctors
  • Analysis of identified discrepancies between the service and doctors
  • • Doctors’ feedback collection and analysis

Constant selective monitoring for technical defects is carried out

Grant payment

Following 3 months of operation as part of the main stage of the Experiment, apply for the grant

The amount of grant to be paid is based on the number of studies analyzed by the AI service

Apply for the next grant payment every 3 months

Questions

Procedure and conditions to conduct the experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images and further use in the healthcare system of Moscow (hereinafter, the Experiment) are regulated by Order of the Moscow Health Care Department No. 43 of January 24, 2020 (hereinafter, the Order ) and Regulation of the Government of Moscow No. 1543-PP of November 21, 2019 (hereinafter, the Regulation )

1
Who can apply for the grant?
As per paragraph 1.1 of the Annex to the Regulation, grants for the use of innovative technologies in the field of computer vision for the analysis of medical images and further use in the healthcare system of Moscow are provided to legal entities that have developed or have the right to provide services based on computer vision technologies intended for the analysis of medical images.
2
What are the documents to be submitted?
Documentation package includes the following:
1) Application filed in accordance with the Annex to the Procedure to Conduct the Experiment (paragraph 3.4 of Annex 1 to the Order ).
2) Explanatory note on the issues set out in Annex 2 to the Procedure to Conduct the Experiment (paragraph 3.5 of Annex 1 to the Order) and paragraph 3.3 of the Annex to the Regulation. Download Checklist of Submitted Documents
1. Service name.
2. Legal grounds to provide the service.
3. Brief information about the service.
4. Certified copies of the legal entity's statutory documents.
5. Valid certificate of conformity of the quality management system to the ISO standard (if available).
6. Certified copies of the state certification as a medical device (if available).
7. Certified copies of certificates of quality compliance of the proposed service with the local standards of FDA, CE, and their equivalents in other countries (if available).
8. Copies of scientific articles on evaluating the accuracy and efficacy of the proposed service published in peer-reviewed journals indexed in Scopus and/or Web of Science (if available, provide references).
9. Originals or copies of reports on clinical implementation or application of the proposed service in medical centers of the Russian Federation or other countries.
10. Duration of acceptance testing of the proposed service in medical centers of the Russian Federation or other countries.
11. The number of and information on the medical centers where the acceptance testing of the service was carried out.
12. Documents confirming the right to provide the proposed service based on computer vision for the analysis of medical images as well as describing the functions, technical architecture and characteristics, intended use scenarios of the service, and its hardware and software requirements.
13. Report on preliminary clinical and technical tests performed in accordance with Guidelines No. 43 "Clinical Acceptance of Software Based on Artificial Intelligence Technologies (Radiology)" (recommended by the Expert Council for Science of the Moscow Health Care Department, Protocol No. 8 of June 25, 2019) (if completed successfully, specify information on the place and date of the tests and attach a copy of the test report). [ссылка на МР]
14. Availability and quality of implementation in the proposed service of the functionality of an automated analysis of medical images, indicating localization of pathological findings detected by the service and providing notification of the results (if available, attach a screenshot as an example)
15. Availability and quality of implementation in the proposed service of the possibility to form a DICOM series as a result of analysis, which contains information on the presence or absence of pathologies and analysis results visualizing the findings on the original images (if available, attach a screenshot as an example).
16. Support of the HealthLevel7 (HL7)/FHIR standard by the proposed service.
17. Availability and quality of implementation in the proposed service of the possibility to automatically provide information to prioritize studies in the radiologist worklist (if available, attach a screenshot as an example).
18. Availability and quality of implementation in the proposed service of the functionality of comparative analysis of studies of one patient performed at different times (evaluating the dynamics of pathological processes) (if available, attach a screenshot as an example).
19. Availability and quality of implementation in the proposed service of the functionality of automated preparation of the draft radiology report (if available, attach a screenshot as an example).
20. The time required to analyze one study using the proposed service.
21. Availability of the results of testing the diagnostic accuracy of the proposed service on depersonalized medical images of citizens of the Russian Federation and/or European and Asian subjects.
22. Diagnostic accuracy metrics of the proposed service, including classic ROC curve, area under the curve (AUC).
23. Diagnostic accuracy metrics of the proposed service, including false-negative rate.
24. Diagnostic accuracy metrics of the proposed service, including false-positive rate.
25. Diagnostic accuracy metrics of the proposed service, including other diagnostic accuracy parameters as per Guidelines No.43.
26. Other documents that the applicant considers necessary to provide in order to receive the grant.
3
Where to apply?
As per paragraph 3.4 of Annex 1 to the Order, the application, alongside with the documents enclosed to it, is submitted to and registered by the Moscow Health Care Department. more details...

Applications are accepted at 43 Oruzheinyi Lane, bld. 1, 1st floor, correspondence reception desk

Opening hours:
Mon, Thu: 8:00 AM to 05:00 PM
Fri: 8:00 AM to 03:45 PM
Lunch break: 12:30 to 01:15 PM
Sat, Sun: closed
4
By whom and how long is the application reviewed?
Grant applications from legal entities are reviewed by the Commission appointed by the Moscow Health Care Department (paragraph 3 of Annex 1 to the Order). As per paragraphs 3.1 and 3.2 of the Annex to the Regulation, the deadline for application review by the Commission does not exceed 10 calendar days from the date of receipt of the application by the Commission.
5
How many applications can be submitted?
As per paragraph 3.6 of Annex 1 to the Order, no more than one application can be submitted from each company for each imaging modality. To simultaneously participate in the selection process by several imaging modalities, the applicant should submit separate applications for each imaging modality.
6
What are the requirements for completing the application?
As per paragraph 3.2. of the Order, the application and the documents enclosed to it should be submitted in Russian on paper, should be bound, numbered, and signed by an authorized person and stamped by the applicant. Corrections and erasures are not allowed. If the original document is issued in a foreign language, it should be accompanied by a translation in Russian certified by an authorized person and stamped by the applicant.
As per paragraph 3.3. of the Order, it is recommended to attach a copy of the application and copies of the documents enclosed to it in electronic form in pdf format on optical media.
As per paragraph 3.8. of the Order, the copies of the submitted documents (other than notarized ones) should be signed by an authorized person and stamped by the grant applicant.
7
In what form should the results of preliminary clinical and technical tests be provided?
In accordance with Guidelines No. 43 (hereinafter, the Guidelines), preliminary clinical and technical tests (hereinafter, PCTT) are conducted under a simplified and adapted clinical test methodology to gain additional knowledge about the performance and limitations of the AI-based software. PCTT do not represent a substitution or equivalent to clinical tests and can include stages of analytical validation and clinical acceptance, or consist of analytical validation alone. The PCTT protocol must include 6 test stages as shown in Table 2 of the Guidelines. The report form is given in Annex 7 to the Guidelines.
8
How are the grant applications reviewed?
The applications are reviewed in accordance with paragraph 3 of the Annex to the Regulation and paragraphs 1, 2, and 3 of the Annex to the Resolution on the Commission of the Moscow Health Care Department to conduct the experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images and further use in the healthcare system of Moscow, which is part of the Order.
As per paragraph 3.4 of the Annex to the Regulation, the Commission forms a consolidated list of grant applicants and conclusions based on application review results.
9
Who makes the decision on grant allocation, what is the procedure for grant provision?
As per paragraph 3.5. of the Annex to the Regulation of the Moscow Health Care Department (hereinafter, the Department), based on the conclusions made by the Commission upon application review, the Department makes a decision on grant allocation, confirms the grant recipient status and the amounts of grants provided by adopting a legal act, and publishes the corresponding information on the official website of the Department on the Internet.
As per paragraph 3.6. of the Annex to the Regulation, grants are provided by the Department based on a grant contract (agreement) concluded between the grant recipient and the Department, which regulates the procedure and deadlines for the grant transfer.
As per paragraph 4 of Annex 1 to the Order, a legal entity that is a party to the agreement concluded to provide the grant in the form of a subsidy from the Moscow city budget is a participant in the Experiment.
10
How is the grant paid?
As per paragraph 1.5 of the agreement to provide the grant in the form of a subsidy from the Moscow city budget to the participant in the experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images and further use in the healthcare system of Moscow, which is part of the Order (Annex 4 to the Order), within 5 working days after issuing the grant payment order, the Competent Authority sends the Recipient an additional subsidy agreement, specifying the actual number of analyzed studies and the total amount of Subsidy to be paid.
As per paragraphs 1.6 and 1.7 of the above agreement, the Recipient signs the additional agreement within 5 working days and submits it to the Competent Authority. The Subsidy is transferred to the Recipient's account in a lending institution registered in the Russian Federation no later than 5 working days from the date of signing the additional agreement.
11
What documents on technical and functional characteristics and service system requirements should be attached to the application?
As per paragraph 3.5 of Annex 1 to the Order, it is necessary to attach documents confirming the right to provide the proposed service based on computer vision for the analysis of medical images as well as describing the functions, technical architecture and characteristics, intended use scenarios of the service, and its hardware and software requirements.
- List of operational documents;
- Operator's manual;
- Record form;
- Application description;
- Software description;
- Explanatory note.
More details are available at the minimum set of software documentation










For more information or if you have questions